Consensus is boring! Washington still won’t pay big bucks for predictions.

This is a guest post by social data expert Nick Arnett

Think Washington power brokers are ready to pay big bucks for Nate Silver’s predictions? Probably not – his numbers are too good. Yes, too good.

A decade ago, as I was starting one of the first social media analytics firms hoping to sell to politicos, a friend in Congress pointed out that even though politicians have to go outside to see which way the wind is blowing before making a decision, they ultimately only want polls that reinforce their beliefs.

Nancy Pelosi gave a fine example on The Daily Show when she dismissed Silver’s prediction that Republicans would still control the House. “That’s why we have elections,” she said.

Organizations rarely look to outsiders – and too often, even insiders – for accurate predictions, unless it is to shift blame. To the side that’s going to lose (and they all do, eventually), Silver is like a consultant hired to justify a huge investment in a horrible product, after the fact. The last thing the client wants is an accurate forecast.

The media doesn’t want accurate predictions either — consensus is boring. An accurate prediction is only newsworthy if there’s another poll that disagrees. Even more to the point, conflicting polls support conflicting people, whose arguments are the staple of news talk.

As many have already observed about Silver, the real sin of being repeatedly right is that his predictions rob us of the sense that we have control. Although helplessness is a big source of stress for anyone, it is huge for politicians and other leaders completely incompatible with success. For example, can anybody imagine that Steve Jobs would care what a poll predicted?

Quite the opposite – Jobs hired Trip Hawkins as Apple’s first head of marketing after Hawkins sent him a forecast doubting the size of the market. Jobs rewarded those who challenged his thinking; he had no interest in accurate predictions unless they came from creative thinking. Even then, it was the creativity that mattered. (Until I figured this out about him, I couldn’t understand why Jobs continued to tolerate, even encourage, my own pointed questions and challenges).

Still, Silver’s exceptional results truly matter.  He gives us a much-needed demonstration that the average “rock star data scientist” is just that — average. In a world where what we see, eat and otherwise use or are used by is driven by statistical analyses, our leaders’ hubris is an ever-present temptation for analysts to follow.

The stakes can be very high. Believe me, I know. A company affiliated with the Department of Homeland Security (DHS) bought my startup’s patents and I was told that DHS applied them to “terrorist buzz.” Two years later, Nov. 10, 2004, my niece’s husband, a Marine, was killed in action in Iraq. Since then, I’ve had to wonder if my own inventions were misused to justify the war that killed him.

Keep it up, Nate Silver. We need you because accuracy isn’t always profitable.

Nick Arnett is the Director of Product Management at NetBase. He was a participant in Carnegie Mellon Silicon Valley’s Disaster Management Initiative, working on citizen preparedness topics, especially crowd-sourced crisis mapping and vulnerable populations.

Prior to joining Netbase, he was the founder of Senti-Metrics Partners, a social media monitoring tool acquired by LiveWorld Inc.. 

Follow him on Twitter @NickArnett