Why log data is the real story behind the Tesla/NY Times dust up

Matt Quinn is CTO of TIBCO, the parent company of LogLogic, which provides log management software.

When you think of “Big data,” you rarely think of log data. It just doesn’t have much sex appeal: It’s what IT uses to monitor applications, compliance, and security.

But we got a reminder this week that log data truly is valuable big data.

After New York Times writer John Broder published a review bashing Tesla’s new Model S, Tesla CEO Elon Musk stepped forward Wednesday with what he said was log data generated by the car during Broder’s test drive, and the data clearly refutes the writer’s account.

Log data and log data management make the world a little bit more honest by giving us hard facts about what’s really happening in the moment, at the most granular level of detail imaginable. Beyond the intrinsic value in data-as-it-happens, log data immediately becomes a historical record that can prove or disprove our hunches and, at times, fabrications.

Whether Musk or Broder are giving us the facts isn’t the overarching point. With this story, log data hit the mainstream consciousness and will be an expectation in the future. “Where’s the log data?” is now the question that can close the argument and help us make the decision about what to believe and where to put our resources.

The Tesla case isn’t the first time we’ve seen log data make headlines. It was in the spotlight recently when MIT was accused of releasing its network logs to law enforcement without a subpoena, providing the information used to accuse Aaron Swartz of downloading scientific journals for public release. Swartz’s lawyers worked to suppress the highly detailed logs because they knew these ‘golden records’ were powerful, detailed evidence of what happened. The Swartz case and its unfortunate outcome is a harbinger of what’s to come with log data.

Log data was also a key component of the case against Martha Stewart in 2004. In 2006, Florida Circuit Court Judge Elizabeth Maass, writing on the e-discovery challenge, said that electronic data “… are the modern-day equivalent of the paper trail.”

Those cases have laid the legal groundwork for log data, but there’s a rapidly changing volume, variety, and velocity of data that makes it much harder to keep up with.

We’re globalizing and automating at a rapid rate, leaving the world more vulnerable to problems on a scale we haven’t seen before, like sporadic, vexing outages in enormous data centers, sophisticated digital fraud launched from anywhere in the world, and even computer viruses that can ‘sleep’ for periods of time to avoid detection before waking up and wreaking havoc. But everything, as Musk knows well as Broder found out, leaves a data trail.

Managing log data is big data without the hype of needles in haystacks. Log data used on the fly stops fraudulent credit card transactions before they happen for retailers like RadioShack, brings up another server before the website goes down for Hamleys in the UK, and demonstrates that we’re following regulations at the detailed level for headset manufacturer Plantronics. Log data is the cold, hard, fast-moving, fast-accumulating truth.

Tesla versus New York Times will fade into history, but the technology Musk is pointing to is the future. This week, the world was introduced to a key big data story whether they knew it or not.

[Top image credit: Maksim Kabakou /Shutterstock]