big data social TVThis post was written by John Funge, cofounder and CEO of real-time big data processing company BrightContext.

For all the legitimate buzz about social TV, one mystery remains: why hasn’t it come about? Why are most social TV experiences basically Twitter feeds?

The answer has to do with a modern problem of “real-time big data”—which is to say, instantly consuming, routing, processing and broadcasting large volumes of data.

Have you ever considered how Twitter sends out 28 million messages almost instantly when Lady Gaga blows a kiss to her followers? Only a small collection of folks that rarely leave their desks at Twitter or the CIA can answer this question. But the answer is central to the future of social TV.

For the past 80 years TV has been a one-way broadcast communication. But the promise of social TV is that the audience will engage with each other (and with advertisers and show producers). Television won’t remain a one-way medium.

This social TV vision is irresistible. But how do you handle the potential real-time traffic load? Just as airports become overwhelmed at Thanksgiving, it’s the same with servers. For load-handling purposes, how many users you get per month matters less than how many you get per minute.

It’s easy to forget just how many people TV draws together at one time.  A rerun of The Big Bang Theory (not to mention events like the Super Bowl) can fill 30 Yankee stadiums of people all watching the same thing together.

For years, TV networks have dealt with traffic spikes on their web sites by using caching services like Akamai. But caching doesn’t work well with content that changes frequently. If you want the audience to hang out and participate for the length of a show –clicking, voting and commenting—you have a problem.

If two million people are watching a Two and Half Men rerun and 0.5% tap into a companion app, that’s like running a chat room with 10,000 people. The numbers will only increase as social TV becomes mainstream.

So why not let Facebook and Twitter deal with all this real-time data? To date, TV shows have done exactly that. Notwithstanding a few apps like GetGlue, social TV as we know it in 2012 is largely Twitter. In 2010, MTV set the bar with its excellent Twitter Tracker for the VMA Awards.  Since then, we’ve seen ever more variations on the concept. In fact, Twitter itself joined the trend–helping curate large live events including a partnership with NBC for the Olympics.

But social TV must move beyond Facebook and Twitter for two reasons. First, in order to excite audiences and remain competitive over time, social TV apps must become more ambitious, more differentiated, more tightly woven together with the linear content, more game-like, more creatively sponsored and more central to the business case for the TV show itself.

Second, as social TV becomes more integral to the business of television, social networks may represent more of a competitor than partner. The stakes are high as the market sorts out who will own these new direct relationships with the audience. Already, Twitter and Facebook have better insight into TV audiences than the programmers and distributors themselves. To unlock social TV’s full potential and preserve independence, app developers will need a new way to handle the torrential load of real-time data.

There are two reasons for hope, both of which have arrived only in the past few years: first, with cloud computing, engineers can deploy server capacity without needing to actually install physical servers into a data center. But cloud computing doesn’t help if the servers can’t work together. Luckily, new database approaches called “NoSQL” now make it possible to distribute database load across many more servers. In combination with cloud computing, NoSQL databases can handle the extreme levels of database write activity that a TV audience can generate.

The bottom line is this: it’s no longer crazy to imagine building a system that enables a million concurrent users to interact in real-time. That’s a big deal. As technical considerations fade into the background, the most creative producers will take social TV in directions we can barely imagine.

John Funge is a serial entrepreneur and founder of Brightcontext, a cloud-based platform that powers real-time, data stream processing. He has experience working in the TV broadcast industry and sold his last company to Scripps Networks, the company behind HGTV, Food Network, and Travel Channel. Funge also advises and invests in early-stage Internet and digital media ventures.

[Top image credit: Angela Waye/Shutterstock]