Business

Here’s what big data can’t do: Chess, weather, and politics

Image Credit: Michael O'Donnell

REDWOOD CITY, Calif. — Do you design by data, or design by your own intuition? Or somewhere in between?

At today’s DataBeat/Data Science Summit, Quid CTO and Oxford physics Ph.D Sean Gourley touched on the balance between and dangers of data-based and intuition-based decision-making.

“When we think about what data can do, it’s important to help people understand the world around them,” Gourley said. “If you’re building systems to make changes in the world, understanding is really core to that.”

In terms of algorithms versus intuition, he continued, “Where does big data leave us as humans? Are we out of the game?

“Biological limitations mean we can’t think quickly enough … but we also know that if you look at a system like weather prediction, we can track data from the last few years, and computers can play a massive role in prediction. But meteorologists adjust the outcomes of the models and give it a 16 percent lift by interacting with that model.”

The conclusion Gourley has come to, he said, is, “Humans and machines perform better than machines alone. … Big data can’t predict everything.”

When it comes to chess, Twitter, or world peace, Gourley said it’s “naive” to imagine that algorithms don’t have any limitations. For example, there’s the issue of data quality.

“Data is messy. You start to understand that the information out there is more than any organization can collect on its own. … There’s huge amounts of information owned by publishers hidden behind paywalls. But if you make all data free, who’s producing it?”

Case in point: Wikipedia.

On the other side, human biases also get in the way of complex problem-solving, especially for topics like geopolitics or casualties of war.

Ultimately, he concluded, practice with arranging that balance will lead to near-perfection.

“The technology we create will modify us. And we have to be okay with that. But we should design it to change us in a way that’s beneficial and will make us smarter, designing it with humans in mind. How do you project this data in a way that can be comprehended?”

Stay tuned for more from the Data Science Summit today in Redwood City, Calif.

More information:

Quid software is used by decision-makers running companies, NGOs, banks, and funds. It captures data, structures it, and enables people to visualize and interact with the information, to understand the global technology landscape. ... read more »

Richard Snee is chief marketing officer at Pivotal. He is also cofounder and advisor at Data Science Central, an online community for Big Data practitioners. Prior to Pivotal, Snee served as vice president of marketing at Greenplum,... read more »

Originally from New Zealand and am now based in San Francisco where I split my time between Mathematical research and my venture backed startup Quid. I have a PhD in physics from Oxford where I studied on a Rhodes Scholarship. My acade... read more »

Powered by VBProfiles


VentureBeat is studying mobile marketing automation. Chime in, and we’ll share the data.
0 comments