This is a guest post by enterprise technology executive Jeff Carr

“Big data” is without a doubt the hottest trend in technology today, possibly surpassing social media, which has held the tech hype crown for years. At its broadest, the definition of big data includes any aspect of harnessing, analyzing and monetizing the massive amounts of data being generated by web and mobile based applications. The sheer scale of the data being generated dwarfs what was considered ‘large’ amounts of data as recently as ten years ago, and all indications are that this trend line will continue.

Most observers would agree that the era of big data started around 2007, when Google’s MapReduce programming framework was integrated with Apache Hadoop, an open source project founded a couple of years earlier to help developers efficiently and cheaply process large amounts of data. Used together, Hadoop and MapReduce made it faster, easier and cheaper to process and analyze massive volumes ofdata than ever before.

At this juncture companies started adopting various forms of Hadoop/MapReduce to capture and filter their data. Companies like Yahoo and later Facebook were some of the earliest to announce petabyte stores of data in Hadoop.

Rapid commercialization of the Hadoop ecosystem, however, has only occurred in the last two or three years, as the revenue opportunity began to reveal itself. As with any big trend in technology, including the RDBMS/client server, internet, and web security trends that preceded it, big data has correspondingly evolved into the technological equivalent of a gold rush. Hundreds of companies have entered the fray with hopes to quickly cash in.

The majority of these companies, which include pre-big data enterprise technology incumbents and a slew of data-focused technology startups, are positioning themselves as the suppliers to the miners of big data. Instead of picks, axes and gold pans, they supply the tools, technologies and services that will help companies monetize huge amounts of data. Needless to say, there is a lot of data to be mined and a lot of money to be made.

A slightly closer look at the big data market demonstrates an obvious, yet often overlooked, truth about where we are in the big data innovation and maturity cycle. Most big data products available today are UI’s, management tools and integration tools focused around open source projects being developed within the Apache Hadoop ecosystem, including Hive, PIG and Zookeeper. Big data revenue, on the other hand, is being driven primarily from services that help design, architect and implement big data solutions using the Hadoop ecosystem. In fact, many of the largest and fastest growing companies in big data today are pure services companies (Opera Solutions and Think Big Analytics come to mind).

I’m not suggesting that there is anything wrong with a services approach to the market. Most of these companies are providing solid value for their customers, which can translate into healthy revenue streams. It does help, however, to have some historical perspective to understand where we are in the big data innovation cycle, and what comes next.

Open source services and support as a primary revenue stream originated in the 90’s when there was a similar gold rush around the commercialization of Linux. Early companies such as VA Linux and Red Hat capitalized on this. Nearly 20 years after the open source movement started, however, there is exactly one company with “pure” open source roots that has more than $1B in annual revenue, and they achieved this in 2012. By contrast, there are many billion dollar technology companies that have innovated new IP-based solutions that monetize major technology trends.

In that sense, it’s clear that we remain in the earliest days of the big data movement. Larger companies looking to monetize their big data assets lack the expertise and know-how to do so, and they are turning to services companies to help them bridge those gaps. As the market evolves, so too will labor skills democratize and broader product innovation begin to take hold, creating less reliance on services-centric companies. To relate this back to my gold rush analogy, the early winners were people selling tools and mining expertise, but the long term winners were the people that actually found the gold!

So what’s next for big data? For any developer team that has felt the pain of building a big data infrastructure, one clear next step is simplification. The diagram below is a basic flow most companies go through to leverage big data:

Building this solution requires a small army of vendors and consultants to combine solutions and technologies in various ways to analyze and (hopefully) monetize their data. It takes months, and in some cases, years. It’s expensive. In short, it’s a pain in the butt, and the result often does not help monetize big data directly, it’s just the first step in the process.

Yet, no one can argue that this is not where the “action” is in big data today.

Simply put, the current state of big data is great for service vendors, and not always so great for big data buyers. It’s a market ripe for innovation. In the immediate future, we can expect an increasing number of product-centric companies to begin to disrupt the patchwork of services-centric solutions that currently exist.

In summary, the “secret” of Big Data is that today it suffers from a dearth of expertise, so the majority of the revenue is coming from a services centric approach combined with open source technologies. I am in no way diminishing the importance and value of open source projects like Hadoop. To the contrary, I’m a huge supporter, always have been.

What I’m pointing out is that the market will evolve beyond an open source, services-driven revenue model when companies begin developing highly disruptive technologies that solve the hardest problems of big data. While open source solutions like those from Apache may play a role in this, history indicates that the most innovation will come from companies engineering entirely new ways to solve the most difficult problems.

Jeff Carr is COO of Precog. Precog is a data science platform designed for developers and data scientists to turn data assets into data-driven features and products inside an application.

Jeff has worked in technology for 25 years with a focus on business development, market assessment, strategy and operations. For the past 11 years he has worked exclusively with early stage companies in markets including network security (Vericept, CipherTrust), VOIP (Borderware SIPassure), and Big Data (Precog).

---

HealthBeat 2013 is a new conference showcasing how technology is transforming health care. We'll explore how IT is driving out inefficiencies on the hospital, practice, and patient levels. Check out full event details here, and register here.

---

July 9-10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Yesterday, I pulled up the YouTube app on my recently updated iOS 6-sporting iPhone 4S. Or, I tried to.

Not finding the video app in my “Watch” folder, where I could have sworn I had put it, I started searching other folders. Not in “Look Up.” Not in “Play” or “Share” or “Navigate.” Finally, I gave up and slow-double-clicked the iPhone’s one button to initiate a search for the app I knew I had.

Source: John Koetsier

No YouTube app in my “Watch” folder

Results: zilch.

Turns out that Apple killed the app silently in the process of upgrading my phone to iOS 6. And I, a long-time Apple user with multiple Apple products and a technology journalist who covers Apple, had missed that little factoid about the latest and greatest mobile Apple operating system.

It also turns out I’m not the only one who has done so.

Ten thousand people have viewed this question on Apple’s forums about the missing YouTube app, and another thousand have viewed this one. Fantine, in Washington, D.C., had an unsatisfactory answer: Go to YouTube in Safari. Noble Seven tried desperately but cluelessly to help, suggesting that people who can’t find YouTube may have turned it off in their iPhones preferences.

But let’s stop and think about this for a moment.

Imagine that this was your Mac (or your PC). And, in an upgrade to a new version of the operating system, Apple or Microsoft deleted applications on your device. Instead of PhotoShop, you now get NewAwesomeImageEditor, and instead of Skype, you get … nothing.

This is outrageous.

Killing the old Google-powered Maps and replacing it with Apple Maps is one thing. It was premature. The app was, if not horrible, at least not ready, and it gave Apple such a black eye that Tim Cook felt the need to issue a public apology — which didn’t translate to restoring the old Maps app — and suggest other companies’ mapping products. (Though not archenemy Google’s, of course.) But at least it replaced something with something, even if inferior.

Killing the YouTube app and replacing it with nothing is even more annoying. But it’s also insulting. And maybe even evil.

A mobile phone is an intensely personal piece of technology. My phone is mine. I bought it, I pay for it, I use it, and I fill it with apps. It travels with me wherever I go, it stays by my bed at night, and it wakes me up in the morning. It tells me what’s happening, it connects me to the world, and it lets me share what I’m doing with those I love. How dare you reach into my phone and delete my apps?

Source: John Koetsier

Guess what the #1 free app by downloads is on the app store?

All of this has been well covered, in a sense.

Ars Technica detailed all the little bits and pieces that Apple has removed from iOS in version six. Others have talked about YouTube’s ungracious dismissal from the iPhone and the opportunity this has provided other video apps.

But no one seems to be talking about the arrogance that it takes to update an operating system and delete apps that users may have enjoyed, become accustomed to, and relied on. Consumers, incidentally, are speaking with their fingers right now: Google’s YouTube is the top free app by downloads in the app store today.

But are we so used to the gated garden of the app store that we don’t even speak out about this anymore? Are we so conditioned to having limited control over the iDevices we purchase that we just take this for granted?

It reminds me of Apple’s Newstand app — the app for magazine subscriptions.

Not only can this app not be deleted, the app cannot even be put into a folder.

Which means that I and millions of other iPhone users are stuck with a user experience of an app that we do not use and do not want … and cannot delete or even file away. (There was a trick to file it in a folder, but that seems to have been disabled in iOS 6.)

This is just unconscionable — even if some fine print in the terms of service or release notes legally cover Apple’s posterior. It’s not the right way to treat users as valued customers, partners, and human beings. It’s not the right way to build user interfaces and experiences. And it’s just plain not right.

Behavior like this may seem minor and inconsequential. But as it continues, and adds up, more people might start to think — and act – like iPhone owner Kaitheloner, who had a different solution in the Apple support forums … one which she gave divine thanks for.

That solution? Go Android:

I thank God that I have an Android tablet (Purchased the unlocked version for a 125$ ) as well as the iPad. Sure the android tablet *****, it hangs like **** sometimes, the interface is bland, but sometimes, it comes in real handy.

And I don’t think that is in Apple’s best interests.

Note: I did ask Apple for comment on this story, but I have not yet received a reply.

photo credit: Dunechaser via photopin cc

Notice I said “wedding” and not “honeymoon.” Sarver has for some time been Twitter’s first line of defense between the company and a occasionally disgruntled army of third-party developers. The relationship between those two entities has been testy at best and an out-and-out war at worst for the better part of two years. Unfortunately, mid-2011 was no time for a honeymoon.

I met with Sarver in Twitter’s San Francisco headquarters, where bikes stood in racks against the walls and neon plastic deer and framed cross stitch decorated the foyer. It’s a hipster paradise, and the clean-cut, blazer-wearing Sarver seemed at once infinitely at home and slightly out of place in it.

But where Sarver is unquestionably beleaguered is in the relatively public forum of Twitter’s ecosystem developers: the men and women who develop apps that use Twitter’s APIs. They try to build businesses that complement Twitter, and when anything goes wrong with that setup (for example, Twitter makes a minor change to an API), Sarver is the de facto shock absorber. And that explains why he skipped his honeymoon to deal with his unruly constituency.

For such a young man, he handles the role of “the guy developers love to hate” with a large measure of grace and resilience.

“The ecosystem is over 750,000 developers,” he told me. “It’s a massive thing. Any community of that size, you’re never going to keep everyone happy at all times. … Some people might be mad at me, but I try to hear what they’re trying to say and fold that back into what we’re trying to do.

“The most critical people are usually the most valuable ones. They tell you where the points of friction are and how you can improve.”

Unlike Sarver, Twitter did get a honeymoon. There were a few years when it was the much-beloved social platform of tech insiders and hipsters everywhere. Developers flocked to it, drawn by its openness, flexibility and simplicity, building a wealth of Twitter apps and integrating it into their own web services.

Now the honeymoon is over. For Twitter to survive challenges from rivals like Facebook and Google+, much depends on whether it can continue to hold onto the love of its developer community.

“The most critical people”

In 2007, the infant microblogging service had yet to establish any real foothold outside San Francisco techies, yet it was trumpeting the wild success of its APIs. Co-founder Biz Stone said at the time, “The API has been arguably the most important, or maybe even inarguably, the most important thing we’ve done with Twitter. It has allowed us, first of all, to keep the service very simple and create a simple API so that developers can build on top of our infrastructure and come up with ideas that are way better than our ideas.”

Yet for many developers, it seemed that as soon as an outside party came up with an innovative idea, Twitter jumped onto the same idea too, stomping on the little guys. A long list of Twitter’s features, including mentions and lists, first appeared as features of third-party Twitter clients, only to be incorporated into the main service.

Of course, since such features are often dictated by user behavior and requests, just how much Twitter was scamming off the ideas of others and how much it was simply responding to user needs are unknown.

What we do know is that the company began to seem opaque to the developers who used its API. By the spring of 2010, when it hosted its first developer conference, developer angst had reached a fever pitch.

As the conference opened, the world learned that Twitter had acquired Tweetie and crowned it the company’s official iPhone app. Other makers of Twitter iPhone apps felt blindsided and betrayed. Businesses went down not with a whimper but a bang. Development shops started wondering aloud what areas of the map were safe territory and what Twitter might decide to annex next for itself. No one seemed to be getting or giving any clear answers.

Twitter had built an ecosystem before it decided on a business model, and while it tried out different avenues and experimented with new features, it wasn’t disclosing all of its roadmap to all of its presumed partners. While some folks benefited from a close relationship with the company and its founders, others suffered from a full-on communication breakdown.

It was a situation that was bound to create bad feelings.

“The change in attitude Twitter had since Chirp conference … is just the feeling that they no longer care about developers or their ecosystem,” said third-party developer Arik Fraimovich in a recent interview. “They seem to look at it from the narrow perspective of ‘what’s in it for Twitter?,’ failing to understand that in an ecosystem, all parts need to contribute and communicate. They also fail to understand that the next creative use of Twitter won’t be developed by their developers, but by some API developer — just as it was since their inception.”

A big part of the problem was what plagues many relationships: Lack of clear communication.

Twitter only gradually worked to clear things up. A year after the Tweetie acquisition, after everyone had adjusted to Twitter’s official, in-house suite of mobile apps, the company made an attempt to communicate part of its roadmap, the upcoming acquisition of TweetDeck and a renewed focus on consumer features.

In a controversial memo to a Google Groups list in March, Sarver laid down the law: Don’t make a Twitter client using Twitter’s API. Don’t duplicate Twitter’s features. Don’t rename them. Don’t channel users away from Twitter’s ads and experience.

It was clear, but it was also surprisingly heavy-handed for a company that until then had been known for its open and easy-going approach.

On one hand, Twitter as a business had every right to control how its APIs were being used. On the other hand, it had willingly and knowingly cultivated a thriving sub-economy of Twitter apps and was now telling many of them to kiss off. It’s a communique Sarver regrets, and it started quite the war of words.

Laura Fitton, the perky, audacious founder of Twitter-related startup Oneforty (and an unofficial team mom for the Twitter app community), told me back in March, “Twitter is full of genuinely earnestly awesome people who want to do the right thing, but it has resolutely failed to create the conditions for real business success on their platform.”

I also spoke to Loic LeMeur, the tall, eccentric French entrepreneur behind the rapidly pivoting Seesmic, who told me, “There are two types of Twitter apps, the ones Twitter likes and the ones that are competitive and don’t have good communication with them.”

Of Sarver’s missive, which put Seesmic in the direct line of fire, LeMeur said, “I wasn’t expecting such dramatic changes, but instead of complaining, I have to adapt to it. … Competing with your number one partner is a little dangerous.”

Jesse Stay is another entrepreneur/engineer who has built a business around a collection of social APIs, including Twitter’s. Stay is the picture of a mild-mannered Mormon, a truly kind man who would think long and hard before uttering a negative word about anyone. Still, during yet another minor but uncommunicated technical change at Twitter, he told me, “This move by Twitter is no surprise, as I’ve experienced over and over through the last three years. It’s why we’ve moved to a $29-per-month plan for all the services we offer. … That’s the only way to afford the risk of Twitter.”

Perhaps no one understands the risk of Twitter better than Noah Everett, a man whose name came up more than once in my talk with Sarver. Everett built TwitPic, a highly popular and successful app for uploading photos and sharing them through Twitter. Out of the blue earlier this year, Twitter announced it would be implementing media-sharing capabilities into its web and mobile apps, first for photos, then for videos, cutting a number of photo-sharing apps — including TwitPic — off at the roots.

“TwitPic will continue to still live on,” Everett told me when the feature rolled out to users, espousing the kind of grim optimism I’ve heard from many entrepreneurs facing hopeless odds. “We’ve got a huge base of loyal users and we still want to continue to provide them with the best service possible … regardless of this feature announcement from Twitter.”

But there was a bitter twist. Everett had the previous year started working on a stealth project called Heello, a project attempting to “make email suck less,” as he put it to me. But out of spite or pique or frustration, he decided to turn Heello into a Twitter clone.

“If Twitter can compete with its developers without fair notice, then why can’t we?” he quipped. “Twitter rolling out their photo option to everyone [the day before the Heello site launched] was a complete coincidence, but I’m glad the timing happened that way.”

These voices and many others ring in my ears – and likely, to some extent, in Sarver’s, too – as we face one another over a small table in a small conference room at Twitter.

I ask him if he thinks the company made any mistakes.

“You can always do things better,” he told me. “If you go back to the guidance we gave last March, we could have executed that message much better. The tone and format could have been improved. But the intent is still important: The most important thing for us as a platform is to give people a clear idea of where we’re going.”

And that clarity of communication is all most of these critical voices were asking for.

Pages: 1 2

The shutdown is due of a multitude of causes. Co-founder Sol Lipman revealed to Mashable that traffic on the site has remained at the same level as it was two years ago, when the service first launched. The costs of maintaining a video-based service are steep, making it hard to justify continuing a service through lean years.

One of the major reasons behind the shutdown, however, seems to be Twitter’s narrowing down of its app ecosystem, which prohibited the startup from propelling ahead. While Twitter has had partnerships with many of its ecosystem developers, specifically with the launch of the new version of its site (many photo and video-sharing sites, for example, for their embedded content), a number of players — including 12seconds — have been left out of the action.

Partnerships with Twitter — and even many of its apps on mobile devices — help services building on the platform to find new markets and audiences. But as this ecosystem tightens and leaves certain players and up-and-comers on the sidelines, Lipman sees the opportunity for developers to build on the platform shrinking.

“I don’t really see the growth in Twitter that we were seeing. I don’t see it as an explosive growth opportunity right now for a third-party application builder. In any ecosystem, eventually there are going to be winners and losers. I think that’s going to shake out more and more as time goes by,” Lipman told Mashable.

On the bright side, two of the co-founders of 12seconds.tv have found jobs at AOL, and another at eBay, where the team will hopefully continue to work on bigger and better future endeavors.