Business

Here’s what big data can’t do: Chess, weather, and politics

Image Credit: Michael O'Donnell
NOTE: GrowthBeat tickets go up $200 this Friday at 5pm Pacific. VentureBeat is gathering the best and brightest in modern digital marketing to help declutter the landscape, simplify the functions, clarify the goals, and point the way to success. Get the full scoop here, and register by Friday to save!

REDWOOD CITY, Calif. — Do you design by data, or design by your own intuition? Or somewhere in between?

At today’s DataBeat/Data Science Summit, Quid CTO and Oxford physics Ph.D Sean Gourley touched on the balance between and dangers of data-based and intuition-based decision-making.

“When we think about what data can do, it’s important to help people understand the world around them,” Gourley said. “If you’re building systems to make changes in the world, understanding is really core to that.”

In terms of algorithms versus intuition, he continued, “Where does big data leave us as humans? Are we out of the game?

“Biological limitations mean we can’t think quickly enough … but we also know that if you look at a system like weather prediction, we can track data from the last few years, and computers can play a massive role in prediction. But meteorologists adjust the outcomes of the models and give it a 16 percent lift by interacting with that model.”

The conclusion Gourley has come to, he said, is, “Humans and machines perform better than machines alone. … Big data can’t predict everything.”

When it comes to chess, Twitter, or world peace, Gourley said it’s “naive” to imagine that algorithms don’t have any limitations. For example, there’s the issue of data quality.

“Data is messy. You start to understand that the information out there is more than any organization can collect on its own. … There’s huge amounts of information owned by publishers hidden behind paywalls. But if you make all data free, who’s producing it?”

Case in point: Wikipedia.

On the other side, human biases also get in the way of complex problem-solving, especially for topics like geopolitics or casualties of war.

Ultimately, he concluded, practice with arranging that balance will lead to near-perfection.

“The technology we create will modify us. And we have to be okay with that. But we should design it to change us in a way that’s beneficial and will make us smarter, designing it with humans in mind. How do you project this data in a way that can be comprehended?”

Stay tuned for more from the Data Science Summit today in Redwood City, Calif.

More about the companies and people from this article:

Quid software is used by decision-makers running companies, NGOs, banks, and funds. It captures data, structures it, and enables people to visualize and interact with the information, to understand the global technology landscape. ... read more »

Richard Snee is chief marketing officer at Pivotal. He is also cofounder and advisor at Data Science Central, an online community for Big Data practitioners. Prior to Pivotal, Snee served as vice president of marketing at Greenplum,... read more »

Originally from New Zealand and am now based in San Francisco where I split my time between Mathematical research and my venture backed startup Quid. I have a PhD in physics from Oxford where I studied on a Rhodes Scholarship. My acade... read more »

Powered by VBProfiles


We're studying digital marketing compensation: how much companies pay CMOs, CDOs, VPs of marketing, and more, with ChiefDigitalOfficer. Help us out by filling out the survey, and we'll share the results with you.