EarthRisk Technologies has developed a new model for predicting extreme weather events. The model identifies weather patterns based on over 82 billion calculations and 60 years of data. It then compares those patterns to current conditions and uses predictive analytics to predict the weather up to 40 days in advance.

The technology is derived from research at the University of California at San Diego’s Scripps Institution of Oceanography. Cofounder and CEO John Plavan said the old standard for weather prediction is built on subjective forecast models that are not accurate beyond a week.

“Hundreds of thousands of atmospheric variables are changing constantly around the globe and the old models aren’t robust enough to take these into account,” Plavan said in an exclusive interview with VentureBeat. “If there is a change to the initial conditions, the whole thing breaks down. We use statistical relationships to predict eventual outcomes and this technique is not subject to the same chaos. We are applying analytics to an industry that is begging for reinvention.”

EarthRisk has collected data from the U.S. and U.K. governments as well as observational data from thousands of scientists and researchers working in the field and the database is updated every day. EarthRisk’s engine searches for correlations and patterns of “statistical significance” and generates forecast probabilities based on this information. The approach uses the past to predict the future.

“Utilities corporations, energy traders, and energy producers are majorly impacted by big temperature changes and spend hundreds of millions of dollars trying to predict them,” Plavan said. “If they know there will be an extreme cold event a month from now, they can use that data to make an actionable decision, and these guys will do anything to gain a small edge.”

EarthRisk’s flagship product TempRisk is the first commercial application of this technology and is geared towards the research and energy trading communities. The company has been developing, refining, and testing the technology for a few years and now plans to expand the business dramatically and explore more commercialization opportunities. There could be more consumer-focused applications down the road, like the ability to check weather in a tropical location before booking a vacation.

---

HealthBeat 2013 is a new conference showcasing how technology is transforming health care. We'll explore how IT is driving out inefficiencies on the hospital, practice, and patient levels. Check out full event details here, and register here.

---

Imagine this scenario: A dinner guest to The Cheesecake Factory in Louisville, Ky. informs his waiter about some funky tasting ketchup. Simultaneously, across the country in Palo Alto, Calif., a customer complains about the color and consistency of the ketchup on his burger.

Are these two scenarios related and potentially linked back to a bad batch from a supplier? And if so, will The Cheesecake Factory be able to prevent such incidences from occurring?

“You need to take structured data like a restaurant’s location and combine it with unstructured data like the color of the mustard or taste of the ketchup,” said Paul Chang, a program director for the consumer products team at IBM.

For restaurant chains with dozens of locations and hundreds of suppliers, it’s a near impossible task to maintain the consistency of ingredients. One screw up from a supplier and they risk unhappy customers, or worse still, a rogue meatball infected with horse meat. In the aftermath of last week’s scandal, Ikea withdrew meat from hundreds of stores to offset the public relations nightmare. (Update: See below.)

To maintain a watchful eye over its supply chain, The Cheesecake Factory is using IBM‘s tools to analyze large data sets from its 175 locations across the U.S.

In future, the restaurant giant may even be able to predict a potentially bad batch (a trigger is set off, like a supplier’s tomatoes are cheaper than usual) before it makes its way to kitchens — and eventually to your table.

“If that mustard didn’t come out the right color because it wasn’t mixed properly on the production line, it’ll be flagged” said Angela Nardone, chairman and chief innovation officer of N2N Global, a provider of software and services to companies in the food supply chain, in an interview.

N2N has teamed up with IBM to provide The Cheesecake Factory with a technology that can communicate critical supply chain data instantly, so thousands of food items won’t need to be recalled and tested. Nardone said they have initiated a conversation with the Centers for Disease Control and Prevention, as it may be easier to track the culprit if a food-related scandal occurs.

It’s a big customer win for IBM, which is pushing its professional services branch into the growing market for “big data” technology. Projections show that the overall market could increase from $8.1 billion last year to $23.7 billion by 2016, according to to the technology research firm IDC.

In February, IBM announced it would cut the price on its least-expensive Power server computers by 50 percent, to under $6,000, in a bid to compete with other large vendors like Oracle and SAP.

The restaurant industry represents a massive opportunity for IBM and startups like Food Genius, which provide chefs with insights into the optimal menu, including what dishes to make, and how to market, and price them.

According to the National Restaurant Association, restaurant sales top $660 billion, and with nearly a million eateries in the United States alone.

Update 3:30pm Pacific: IBM’s technology does not actually help restaurant chains detect horse meat, a test that requires DNA testing. IBM provided the following statement to VentureBeat:

While there are certainly applications for this technology beyond brand consistency, including food tracking and sourcing, they are currently using IBM’s tech over at Cheesecake Factory to ensure a consistent food color, taste and texture.

photo credit: Simon Greig (xrrr) via photopin cc

---

Big Data and Predictive/Real-time Analytics startups: Are you looking to jumpstart development & accelerate market traction? Sign up for the SAP Startup Focus program to receive technology, support, resources and community to help you develop new applications on SAP HANA, a cutting edge database platform. Get started here, and enter promo code “VB2013″ on the form.

---

This is a guest post by technology executive John De Goes

“Big data” is dead. Vendors killed it. Well, industry leaders helped, and the media got the ball rolling, but vendors hold the most responsibility for the painful, lingering death of one of the most overhyped and poorly understood terms since the phrase “cloud computing.”

Any established vendor offering a storage or analytics product for a tiny or a large amount of data is now branded as big data, even if their technology is exactly the same as it was 5 years ago (thank you, marketing departments!). Startups, too, lay claim to the moniker of “big data app” or “big data startup,” eager to soak up some of the big data money floating around in big data-focused VC funds.

The phrase “big data” is now beyond completely meaningless. For those of us who have been in the industry long enough, the mere mention of the phrase is enough to induce a big data headache — please pass the big data Advil. (Editor’s note: We couldn’t agree more!)

If you want proof, witness the rising tide of backlash against the term:

• “Every so often a term becomes so beloved by media that it moves from ‘instructive’ to ‘hackneyed’ to ‘worthless,’ and Big Data is one of those terms….” Roger Ehrenberg
• “Every product by every vendor supports big data… and every ‘industry leader’ with every talk needs to include the phrase in the title of their talk and repeat it as many times as possible. So every data warehouse pitch is rehashed as a big data pitch, every data governance, master data management, OLAP, data mining, everything is now big data.” Rob Klopp
• “Big data as a technological category is becoming an increasingly meaningless name.” Barry Devlin

RIP Big Data.

Now that big data is dead, we’re free to move onto the next chapter of our lives. Which, from a data perspective, means we can stop worrying about the volume, variety, velocity, veracity and verisimilitude of data (just put it in Hadoop already!), and begin focusing on ways to impact bottom-line metrics by leveraging the talent, tools, and technologies that are slowly making their way into mainstream.

As the industry matures, there won’t be a single term that replaces the big data moniker. Instead, different tools and technologies will carve out different niches, each more narrowly focused and highly specialized than the universal sledgehammer that was big data.

I’m going to talk about some of the niches you’re going to hear about again and again. Alas, some of these will be spun into buzzwords that, like big data, accumulate so much “momentum” they eventually lose meaning. But for now, they should give you a glimpse into what lies ahead in the future of data storage, processing and analysis.

Meaningful jargon for 2013

I’ve identified six different aspects of data that you’re going to hear more frequently in 2013. Each of these terms actually conveys useful information, and cuts across slices and use cases that fall under the rubric of “big data”.

Smart Data?

Various industry leaders, writers, speakers, and pollsters (see here, here, here, here, here, here, and here) have started using the term “smart data” to refer to an increasingly common pattern emerging in the big data scene involving the productization of persistent data through predictive analytics.

In essence, companies are moving beyond BI, which relies on humans to interpret data, and are looking to monetize their vast troves of machine-captured data through predictive analytics (which relies on advanced techniques in statistics and machine learning to recognize and exploit patterns). These predictive analytics are often deployed as revenue-generating, intelligent features inside products, such as fraud detection, recommendations, personalizations, ad targeting, and much more. Examples of companies leveraging smart data include Netflix, Amazon, Rich Relevance, Gravity, LinkedIn, SailThru, and many more.

Data Science?

Data science is a new field that employs advanced techniques in statistics, machine learning, natural language processing, and computer science to extract meaning from large amounts of data (sometimes with the goal of creating new data products — arguably the reason data science was created). Though still meaningful, this term is starting to come under abuse from vendors due to its skyrocketing popularity. Metamarkets, for example, touts the benefits of its “data science platform”, but their core technology is a slice & dice aggregator. Similarly, many people who know SQL and MicroStrategy are now claiming to be data scientists. I fear this term may become a victim of its own success and suffer the same fate as big data.

NewSQL?

NewSQL is a moniker for describing highly-scalable, horizontally-distributed SQL systems. Drawntoscale, VoltDB, SpliceMachine, SQLFire, Impala, Redshift, Clustrix, NuoDB, and Hadapt are a few of the many solutions that combine the scalability of NoSQL platforms with SQL and the strong ACID guarantees of legacy relational databases. NewSQL doesn’t mean NoSQL will die, it just means that companies who want scalability and SQL can have their cake and eat it, too.

Many companies will continue choosing NoSQL systems because they support non-relational data and can offer higher performance because they don’t provide ACID guarantees.

Predictive Analytics?

After many years of relative obscurity, predictive analytics are coming into their own. Core to both data science and smart data, predictive analytics are the flip side to historical analytics, and involve using historical data to predict future events. If you can predict the future, you can also change it.

Indeed, predictive analytics are behind everything from recommendation engines (which recommend items that are predicted to maximize the chance of a conversion), to fraud detection, to yes, predicting which parolees are most likely to commit murder. The field calls upon techniques in statistics, machine learning, modeling, and other fields to identify and exploit patterns.

Trends that didn’t make the cut, but are worth paying attention to, include stream processing and streaming analytics, NLP (which seems well on its way to entering mainstream, thanks in no small part to technology vendors like AlchemyAPI), image and video mining (including face, gesture, and emotion detection), machine learning, in-memory storage and computing grids, and graph databases, which offer a completely different way of solving problems in data analysis.

Life after Big Data

Big data as a term has seen its heyday. While many of the challenges that gave rise to the term are valid, storing virtually infinite amounts of multi-structured data is no longer novel or even mildly interesting.

Moreover, widespread and proliferating abuse of the term by vendors means that it means less and less with each passing month.

Increasing sophistication in the storage, processing, and use of data means we’re probably not going to see a single term replace big data. Instead, we’ll see the most common use cases forge on ahead, adopting terminology more restrictive and more descriptive.

Welcome to the post-big data era! It’s going to be one hell of a ride.

John A. De Goes is CEO of Precog, a company focused on making it easy to develop and deploy advanced analytics. He founded Precog after struggling with big data challenges as VP Engineering at LivingSocial. 

De Goes is an established and best-selling technical author and a major open source contributor. He has been working in distributed systems design and development for more than a decade.

Frustrated journalist receiving another “big data” pitch // Joshua Minso, Shutterstock 

---

HealthBeat 2013 is a new conference showcasing how technology is transforming health care. We'll explore how IT is driving out inefficiencies on the hospital, practice, and patient levels. Check out full event details here, and register here.

---

Won a place at a four-year college? The odds are against you that you’ll graduate in less than six years — if you complete your program at all. There are an estimated 37 million people in America that have some college experience, but no degree.

If there is a technology solution to this problem, Canadian startup Desire2Learn, wants to be the first to market. With its third acquisition, the well-funded ed-tech company is exploring how predictive analytics can reduce the time it takes for students to graduate college.

Jeff McDowell led the acquisition of “Netflix effect” software Degree Compass

Today, the company has bought the much-hyped “Netflix effect” software for an undisclosed sum. The product is called “Degree Compass,” and it is the brainchild of Tristan Denley and a team of developers at Austin Peay State University, who received a $1 million grant from the Gates Foundation.

It works by helping its users select courses based on their academic interests, and the success rates of students with a similar set of competencies. Inspired by sites with recommendations engines like Netflix and Pandora, the algorithm combs through hundreds of thousands of previous students’ grades and transcripts to make its highly personalized suggestions.

Degree Compass pulls its data from the student information system (SIS), so it can be installed in any university, regardless of whether its a customer of D2L’s learning management system.

According to Jeff McDowell, D2L’s business development and marketing lead, this buy-up will create social value and generate revenue for the already profitable company. “The next evolution in this space [ed-tech] is the analytical layer with all the data that is generated about students and schools,” he said by phone.

McDowell said he was recently approached by Denley to discuss ways to commercialize the software (see below), which was already being used by colleges across Tennessee. This morphed into a conversation about a potential acquisition, with Degree Compass’ developers and data scientists joining the D2L team.

Students get course recommendations in the form of star ratings.

D2L has been building its own analytics toolset for teachers and administrators since it launched in 2009, and saw an opportunity to reach students. The company’s software-as-a-service tools are used by 700 K-12 and higher-ed institutions throughout North America.

Degree Compass is likely to be folded into D2L’s existing set of products in the long-term. “I suspect it could happen,” said McDowell, who added that Denley is under contract to provide ongoing “innovation and evangelism” and help the company connect with college students.

“The ROI on this is huge,” he continued. “If the product is able to save two students from dropping out of college, the tool has paid for itself.” McDowell would not reveal pricing information at this time.

In September 2012, we reported that Desire2Learn had received the largest infusion of investment capital in Canadian startup history. The Waterloo, Ontario-based company pulled in $80 million from New Enterprise Associates (NEA) and OMERS Ventures, the venture arm of one of Canada’s largest pension fund managers.

Top image via Lasse Kristensen // Shutterstock 

InsideSales applies data science and statistics to increase sales for businesses. Its products provide cloud-based sales automation and predicative analytics services to get the most out of remote sales efforts and stay on top of leads. Tools include the PowerDialer, PowerText, and PowerSocial, which help salespeople increase their reach, as well as gamification tools and scheduling assistance. It’s PowerDialer for SalesForce 5.0 is one of the most popular applications on the Salesforce AppExchange.

Clients include Cisco, Dell, Eloqua, Gannet, and Groupon. Sales for InsideSales itself grew by over 100 percent last year, according to a statement issued by the company, and are continuing to grow. This hefty investment will help scale the platform as well as add new solutions.

New investor U.S. Venture Partners and existing investor Hummer Winblad Venture Partners led this round.

InsideSales.com was founded in 2004 and is based in Provo, Utah. This brings its total funding to $39 million to date, following a $4 million Series A in August of 2012. Read the press release.

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

It is a big day for precipitation and data. With gloomy skies and CloudBeat 2012 going on right now in Redwood City, the time is ripe for cloud company AgilOne to make two big announcements.

Today, the company launched its flagship solution (also called AgilOne) and announced the closure of a $10 million round of investment.

AgilOne provides cloud-based predictive marketing intelligence to help marketers figure out what their customers are going to do next. The platform serves as a “data scientist in the cloud.” It not only helps teams make sense of large quantities of data, but also make smarter marketing decisions and address issues of acquisition, retention, and reactivation.

Predictive algorithms monitor clients’ sites for risks and opportunities and send alerts about what is happening in the customer base. AgilOne then applies machine learning to show the reasons behind trends and results, and provides dashboards for measuring and analysis. Detailed customer profiles are also available, so marketers can quickly react to their customers’ behavior.

The platform is tailored to high-volume marketers and it works with large corporations such as Bosch, ideeli, Moosejaw, PetCareRx, and Shazam. Many marketing professionals do not have a background in complex math, and AgilOne seeks to give them the information they need without it.

The $10 million in financing was led by Mayfield Fund, with participation from Sequoia Capital who led the $6 million Series A last year. The money will be put towards expanding sales, marketing, and engineering efforts. AgilOne is based in Mountain View, California.

Lattice Engines, which bills itself as “Big Data for Big Sales,” is bulking up with big money. Today the company announced closing a $20 million round with venture capital big whigs New Enterprise Associates and Sequoia.

Lattice’s salesPRISM platform analyzes data and delivers real-time reports with specific and actionable data to sales representatives, who can use the information to generate leads and close deals. Its “secret sauce” (for data, not burgers), is the use of predictive analytics to help sales people anticipate their customers’ behavior.

According to Lattice, the technology can increase pipeline opportunities by up to 75%, triple conversion rates, and double cross-sell rates at existing customers. These results have helped the company double its enterprise customer base, win multiple awards, and work with big organizations like Adobe, ADP, Cox Communications, Dell, Staples and SunTrust.

Lattice is based in San Mateo. This is its third round of institutional funding. Read the press release.

The day before Thanksgiving, Prior Knowledge was gobbled up (gobble gobble) by Salesforce.

Prior Knowledge is the company behind Veritable, a predictive database that provides insight into the process of building applications. Referred to as “infer-structure,” the API examines the underlying infrastructure of an application to find causal relationships. It then tells developers what they might be missing. Using this system, developers without deep knowledge of statistics or analytics can make data-driven predictions.

Since the product entered public beta in July and made it to the finals at TechCrunch Disrupt, there has been a lot of excitement surrounding the technology and its MIT PhD holding founders.  Salesforce has been on an aggressive buying spree this year of mobile, social and cloud startups, and while P(k) does not fit into any of those categories, it could help Salesforce take on enterprise behemoth Oracle. Now, Salesforce has an “oracle” of its own, a database that can predict the future.

With this acquisition, Prior Knowledge will shut down the Veritable API and service on December 5. The details of the purchase have not been disclosed. Read the company’s blog post.  

“Big data” isn’t just an enterprise trend. It’s a technological innovation that is already making a difference in your life.

Police are mixing crime data and sociological information to anticipate incidences of crime. A small cadre of scientists in Silicon Valley is harnessing genetics data to detect early signs of disease. For business owners and harried IT executives, it’s easy feel overwhelmed with the flood of so-called big data options on the market. That is, if you buy into the trend at all.

We believe it’s time to cut through the hype and show you some cool companies that use big data to further research in the fields of healthcare, law, government, and education. To assemble our inaugural list of 10 standout companies, we spoke with investors, analysts, and experts.

We narrowed it down to some ground-breaking favorites who helped define the field. You’ll notice that the companies cover a lot of ground — we anointed a leader in every category, such as data science, business intelligence, data visualization and analytics. We also threw a few impressive newcomers into the mix as well.

Meet our inaugural list of big data companies that are exploiting data in ways that you wouldn’t expect.

Data as a Service/Metamarkets

Michael Driscoll, Metamarkets’ CEO.

It’s been one heck of a year for Metamarkets, a startup that analyses things like tweets, payments, and check-ins for online publishers and web companies to better answer questions like: “Why are customers canceling their memberships?” or “How are users moving through the site?”In the spring, investors clamored to pour money into the San Francisco-based company, which currently has about $23 million in the bank. Aside from being freshly minted with venture funding, here’s why Metamarkets makes our shortlist.

It took guts to down an offer from Twitter: Michael Driscoll, who was Metamarkets’ chief technology officer at the time, wouldn’t back down from his vision to build a bigger company.

It has some cool customers: The technology sees use from digital media companies like the Financial Times and AOL, who need to get better insights on the value of their ad inventory.

“At the customer level, we’ve learned to paint our product in more emotional and less technological terms. Ultimately, big data and its technologies are not the story, it’s about helping people do their job better,” says Driscoll, who is now Metamarkets’ chief executive.

It has one straight-talkin’ CEO: Driscoll told me that we need to reexamine all the hoopla around Hadoop, the open-source computing software that is leveraged by companies like Twitter to store terrabytes of tweets per day: “It’s a service, not a solution,” he said. Driscoll told me he looks up to Aaron Levie, the 20-something CEO of cloud storage startup Box. He told me we need more founders who are willing to bring a natural-born salesman’s brand of personality and smarts to enterprise.

The technology is designed to be easy to use: Metamarkets provides a packaged Software-as-a-Service application so customers can get up and running in a few days. It scales as customer data volumes grow.

Bonus: Don’t miss Driscoll’s appearance at VentureBeat’s upcoming CloudBeat conference.

Data Visualization/Tableau

Christian Chabot, CEO of Tableau

Tableau is a data visualization startup that appears on nearly every list of hot big data startups, but it hasn’t raised a funding round in four years. It is one of the small cadre of companies that are responsible for putting data visualization on the map. Here’s why Tableau made our shortlist.

It doesn’t need your VC cash: The startup’s last funding round in 2008 was for $10 million. It hasn’t taken a dime since then, and it’s on the road to profitability.

It’s changing the way we consume digital media: Tableau is a favorite for media folk experimenting with new ways to use data in their stories. Tableau offers a tool that lets anyone create gorgeous charts and maps for free, no programming expertise required. Reporters at the Wall Street Journal and the Huffington Post use it.

It’s putting data in the hands of people who need it: Big data can be useful for everybody in many ways — it’s not just a money-making tool for businesses. By putting this tool into the hands of a nonprofit like government watchdog California Common Sense (a current customer), Tableau’s software is analyzing volumes of political data to expose fraud and inconsistencies so voters can make informed, data-driven decisions before they vote.

Eat your heart out, Google: “Our mission, to help people everywhere see and understand data, isn’t all that different from Google’s,” Christian Chabot, the company’s cofounder and chief executive, told VentureBeat.

Pages: 1 2 3 4

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

For marketers, it’s not always easy to prove return on investment (ROI), particularly for their social media campaigns.

Anametrix is one of a growing number of startups that provide “big data” analytics tools to marketers. Today, the San Diego-based company has raised $4.4m from TVC Capital in its first round of venture funding.

Anametrix says it wants to help CMOs and lower-level marketing professionals increase their ROI, and better-target their email and social media outreach. The technology collects data from any source such as Facebook, Twitter, LinkedIn or community forums, which is analyzed and visually displayed. The idea is that with a better understanding of the data, marketers can uncover new revenue opportunities.

The company was founded in 2010 by WebSideStory’s analytics team. In 2008, WebSideStory was acquired by Omniture for $394 million.

For startups like Anametrix, there is a huge market opportunity. However, to succeed they will need to displace companies like Tableau that provide data visualization tools for marketers, and Salesforce, which recently unveiled its software suite, Marketing Cloud.

A recent study from Gartner, a research firm, shows that marketers will spend more money on IT than CIOs by 2017.

“Our Series A funding enables us to scale sales and marketing efforts to exploit the increasing market demand for digital analytics and to fuel product innovation, particularly in next-generation predictive analysis for marketers,” the company’s CEO, Blaise Barrelet, told me.

With this funding round, Steve Hamerslag of San Diego-based TVC Capital, will be joining the company’s board of directors.

Image via Shutterstock

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Imagine being able to predict when a war will break out. Well, scientists are trying, and they’re getting awfully close.

In 2010, when the WikiLeaks scandal broke, a group of friends gathered together at Bitly Inc in New York to try to bring order to the chaotic mess of data WikiLeaks had unleashed.

They used a simple code to extract dates and locations from about 77,000 unclassified reports from both simple stop-and-search operations and battles. This revealed unexpected hot spots: the Pakistani border, and the country’s main highway, which had experienced a series of violent outbreaks.

Recent advances in big data technologies have shown that we’re close to being able to predict exactly when a battle will break out. However there’s one thing that will stymie advances in this field: the inherent unpredictability of humans.

This year, when big data became a buzzword, the team reunited at Bitly’s offices to give the project another shot. This time, they teamed up with the brightest mathematical minds for a more audacious goal: a war zone prediction model. The researchers found a general pattern to the violence in Afghanistan, using it to determine whether an uprising would take place in each province, and its level of intensity.

The model worked with surprising accuracy and didn’t fail even when President Obama changed the rules of the game by sending in 30,000 additional troops.

The project, the results of which were published online by the Proceedings of the National Academy of Sciences in July, is just one small part of a growing movement to anticipate episodes of armed conflict using algorithmic computational techniques. Still, we have a long road ahead of us before this data is turned into actionable intelligence — a matter of life or death on the battlefield.

As Lt. Gen. Michael Oates, head of the Joint Improvised Explosive Device Organization, recently stated, “There is no shortage of data. There is a dearth of analysis.”

Iraq: the first ‘big data war’

Bitly’s research wasn’t the first time a group of renegade scientists brought the power of analytics to a war zone. For the better part of the decade, bringing big data to the battlefield has been the job of civilian researchers.

Cast your mind back to the spring of 2003, when four countries participated in the invasion of Iraq and succeeded in toppling the regime of Saddam Hussein in 21 days. At Oxford University, a young scholar had a theory, one that did not sit very well with centuries of political theory. He wondered whether wars share a single, predictable pattern.

Sean Gourley (pictured), a New Zealand-born graduate student, told me that “during one of those classic Oxford dinner conversations where you sit around these high tables, Harry Potter-style”  he butted heads with James Woolsey, the former director of the CIA.

“I had a hunch that there might be some strong, mathematical pattern that might emerge once we’d looked at Iraq,” he said. “No one had really done it before.”

Using a combination of reports from 130 news sources, SMS-based communications between freelance journalists and photographers stationed in Baghdad, plus any information from the frontlines he could get his hands on, he set to work on an algorithm.

“We were writing software to extract when people were dying and how they were dying …”

Gourley told me that he harbored strong reservations about the research. “My God … we were writing software to extract when people were dying and how they were dying,” Gourley said, admitting that he frequently considered throwing in the towel.

To make matters more complicated, reports from various news media often conflicted. As we know now, war reporting is notoriously inaccurate and is even less viable when comparing one armed conflict to another. In Afghanistan, for example, rural environs and a depleted number of reporters on the field led to less coverage than in the Iraq conflict. This remains a problem for researchers today.

Gourley told me they learned that the best approach was for humans and algorithms had to work hand-in-hand. By then, a team of physicists were working with him to see him through this crazy experiment, and they were investigating other conflict areas, including Sierra Leone. According to Gourley, the senior-level military personnel were keeping a watchful eye on their progress.

However, his frequent attempts to convince a contact at the Pentagon to hand over data fell on deaf ears. As a foreign national, he was not able to access official U.S. military reports.

This proved to be a blessing in disguise. At that time, the military’s analysts were typically trained in political science, not computer science, and their reports were spotty at best. “They couldn’t write a python script if you paid them,” Gourley joked. After the WikiLeaks scandal, he discovered that his datasets were superior to the U.S. military’s. “It turns out that we had 80 percent of what they had; they only had 70 percent of what we had,” he explained.

The results, published in Nature in 2009, found that insurgent wars follow an approximate power law, in which the frequency of attacks decreases with increasing attack size to the power of 2.5. That means that for any insurgent war, an attack with 10 casualties is 316 times more likely to occur than one with 100 casualties.

This may seem like a boring set of numbers, but for the first time, it revealed an underlying pattern to war. “It shows that there is something going on in the way these wars are fought that is common to all,” Neil F. Johnson, a physicist at the University of Miami who participated in the research, told Nature.

In the course of his research, he and his team collected data on 54,679 “violent events” reported in nine different conflicts, including those in Iraq, Afghanistan, Peru, and Colombia.

What’s next for the big data wars?

The Gourley and Bitly research are first steps in bringing objective quantitative analysis to realms that were once subjective. Big data will play a growing role in maintaining global security as the Department of Defense reshuffles budgets and priorities. According to Forbes, the amount of data from drones and other surveillance technology has risen 1,600 percent since 9/11.

To step up the research, the U.S. military recently made a $250 million bet on big data. In May, U.S. Secretary of Defense, Leon Panetta put forward a review of the country’s national defense, spotlighting information processing as a growing priority.

For Gourley, who ultimately left academia to form big data startup Quid, this research will have far-reaching consequences, and not just for the military. If these algorithms work, they may change the very nature of war.

Key Image via Shutterstock 

The score is generated from more than 200 terabytes of data, including 7,000 expert reviews and two million user reviews from sites like Amazon and Best Buy. The score is on a scale of 1 to 100 and gets grouped into one of four idiot-proof categories, ranging from “Love It” for high scores to “Don’t Buy It” for bad scores.

Currently, 22,000 electronic products are tagged with a beta version of the score. From this pool, 9% of products have scores 90% or above; 31% products are in the “We Like It” group; 48% of products fall under the “You Can Do Better” heading (ouch); and 12% sit at the bottom of the barrel.

If the stereo system you thought of purchasing is not recommended, Decide.com will present you with higher rated alternatives.

When it launched in 2011, the site offered price predictions for electronics and appliances by analyzing big data to determine whether it was best for a consumer to “buy” or “wait”. Using this algorithm, it has saved consumers an average of $101 per product.

The company is now expanding the price prediction “buy” or “wait” feature to three new categories: sports and outdoors equipment, tools and hardware, and lawn and garden products.

The service was co-founded by Oren Etzioni, a master of price comparison technology. His previous startup, Farecast, applies predictive analytics to airline fares, and another of his startups, Netbot. was the world’s first major comparison-shopping site. Decide.com has raised $8.5M from Madrona Venture Group, where Etzioni is a venture partner, Maveron, and angel investors. It is based in Seattle, Washington.

At a time when technology companies are spending large sums of money battling it out over patents, a new technology promises to let you know your chances of winning a patent suit before you even start.

Lex Machina is a machine learning technology that took Stanford researchers six years to develop, and it brings Big Data to one of the most complex and convoluted areas of the law.

“Lex Machina crawls hundreds of thousands of legal documents to predict outcomes for intellectual property cases,” said Owen Byrd, a spokesperson for the company, in an interview with VentureBeat.

The company just announced that it has raised $2 million in first-round funding, led by Portola Valley’s X/Seed Capital.

The funding announcement comes just days after Kodak lost a landmark suit against Apple and RIM.

Byrd referenced the recent Kodak case to explain how clients are using the technology. The company has data from 130,000 court cases and crawls the Web to extract documents from court records. Byrd told me that if Kodak’s lawyers had used Lex Machina, the technology would have unearthed similar cases, and likely would have found that the digital imaging company would not have won its suit. Alternatively, an analysis of the data may find that companies have a better shot of winning the case in a different state, or with an alternative district judge.

Byrd told me that in future, the technology will expand to other areas of federal law, including antitrust cases, bankruptcy, and tax law. The company will also use the funding to expand its team.

Other participants in this round  include Jeff Hammerbacher, founder of Cloudera, and Naval Ravikant of AngelList.

Click here to read a round-up of Silicon Valley legal technology startups.

Top image based on photographs from Juan Nel and ostill, Shutterstock

Kaggle, a San Francisco-based startup that hosts data science competitions, has uncovered some disconcerting insights about human behavior in its two-year run. At times, its founders have been surprised by the accuracy of an algorithm, and the competitions continue to evoke controversy.

In short, data can be dangerous. I caught up with the company’s founder and CEO, Anthony Goldbloom, to find out more about recent data-driven discoveries that have rocked the boat.

1) “The Essay Scoring Competition”

Sponsor: Hewlett Foundation / Prize: $100,000
Goal: To get the computer to give an essay the same score a human grader would.

The idea was that by analyzing spelling and punctuation, as well as sentence structure, an algorithm could give an essay a reliable score, perhaps even more consistent than a human grader.

Martin O’Leary, a glacier scientist at the University of Michigan, was one of hundreds of competitors from around the world. He told Reuters that he discovered that human graders are rarely in agreement. They are swayed by irrelevant, aesthetic factors like how neatly a student writes. Unlike an algorithm, they award scores that seem random.

“The reality is, humans are not very good at doing this,” said Steve Graham, a Vanderbilt University professor who has researched essay grading techniques in an interview with Reuters.

A controversial study, indeed. Reflecting on the competition, Goldbloom was not initially convinced it could be done. ”I remember thinking: Are we going to be falling flat on our face? It’s really hard to take an essay and give it a grade,” he recalled. The biggest obstacle was to find a team with the requisite machine learning expertise and the ability to deal with unstructured data, including text.

As critics pointed out, it’s easy to outsmart an algorithm and optimize for success. In response to the New York Times’ article which advocated “obfuscating mellifluously” when facing a robograder (i.e. disarm with big words), Goldbloom said the algorithm would need to be refreshed at least once a year.

Key insight: An algorithm is no less reliable at scoring essays than the average teacher. 

2) “The Twitter psychopathy competition”

Sponsor: Online Privacy Foundation / Prize: $1,000
Goal: Can Twitter detect psychopathy? How about personality traits?

In two separate competitions, the goal was to analyze a person’s social media flow to detect personality type and risk for psychopathy. Can we get a sense for your personality in a single tweet?

The data, drawn from the tweets of 3,000 people, may surprise you. “It turns out that with Twitter data alone, we can go quite some way into figuring out someone’s personality,” said Goldbloom. The signals for psychopathy: good grammar, an angry tone, use of swear words and conjunctions.

Before their tweets were analyzed, the users completed a personality test. The algorithm immediately rooted out the most extroverted of the lot. Those who reference others in a reply are more likely to be extroverted. One competitor, a Japanese ad-targeting expert, found that openness is the personality trait that, ironically, is the most difficult to detect.

Are you giving more information about yourself than you realize? Goldbloom said this information could potentially be used by potential employers, as well as for targeted online advertising.

Key insight: With only 140 characters, data scientists and statisticians can get a strong sense for your personality. That’s fairly worrying, considering that this information could get into the wrong hands.

Featured Image via Shutterstock

Dan Johnson, the economics professor known for his remarkably accurate sports predictions, is at it again. Today, the Colorado College professor released his prognostications for the 2012 Olympics Games in London.

Each year, Johnson punches numbers into a computer and out pop the results.  Over the past five Olympics, from the 2000 Summer Games in Sydney through the 2008 Summer Games in Beijing, his model demonstrated 94 percent accuracy between predicted and actual national medal counts.

Dan Johnson, Professor of Economics at Colorado College

Will the United States emerge victorious? It should get 99 Olympic overall medals and 34 gold medals, according to Johnson, which will be the most medals won. Alongside the U.S., China and Russia will round out the top three.

In 2010, he told USA Today that the predictions were an experiment to sees if economic models could be used to predict medals in the same way we could predict unemployment rates. “We thought we’d be wildly wrong,” Johnson recalled. They weren’t.

Johnson’s team uses a handful of economic variables and no athletic data. His factors include per-capita income, the nation’s population, and the home-field advantage for hosting the Games or living nearby.

In a statement released by the university, Johnson said the purpose of the predictions is to provide a benchmark to help set national expectations at realistic levels. For a small nation, winning three medals is an amazing accomplishment. “For the U.S. or Germany or Russia, it’s appropriate to expect a lot more,” he said in the release. “How much more? That’s where the model comes in.”

Curious about how your home country will fare? Read the predictions for 134 competing nations here.

Medals image courtesy of Greater London Authority  

It may seem like everyone on the planet has played Angry Birds, but Rovio‘s mobile game has been downloaded only 300 million times, and the company wants to hit more than a billion users. To that end, Rovio is teaming up today with Medio Systems to take advantage of its predictive analytics service.

The deal isn’t a gigantic one, but it tells you what successful game companies are doing in terms of data mining to extend the reach of successful games. Medio is an analytics vendor that says it goes beyond mere reporting of user data. Rather, it forms a profile of users and makes real-time predictions about the kind of content, offering or monetization offers a user will respond to. Medio will support Rovio titles across all consumer platforms including Google Chrome, Android devices, the iPhone and the iPad.

Rovio’s Angry Birds games are played about 1.4 billion minutes a week, producing a ton of data. Medio taps location-aware connected devices, structured and unstructured data (like numbers or videos), and ways to find more users. The idea is to absorb all of that data and turn it into useful information for improving fan satisfaction, getting users to play longer, and making the right pitches to them.

Robert Lilleness, chief executive of Seattle-based Medio, said the platform was built for the age of Big Data and can scale to handle millions upon millions of users. Medio has been profitable since 2010 and grew revenues 280 percent last year. The platform generates 1.3 billion personalized recommendations each month. Other customers include T-Mobile USA, Verizon Wireless, Telus Mobility, CBS and ABC. The company was founded in 2004 and competes with rivals such as Flurry.

Medio is backed with $46 million in funding to date from Accel Partners ( a Rovio investor), Frazier Technology Ventures, Trilogy Equity Partners, and Mohr Davidow Ventures. In June, Medio announced its analytics platform monitored more than 2.5 million daily unique users. With the addition of Angry Birds, you have to figure it’s going to be more than that now.

---

GamesBeat 2013 is our fifth annual conference on disruption in the video game market. You'll get 360-degree perspectives from top gaming executives, developers, and analysts on what’s to come in the industry. Our theme this year is “The Battle Royal.” Check out full event details here, and grab your early-bird tickets here!

---

This article is brought to you by Thomson Reuters and the Knowledge Effect. To learn more, please click here. As always, VentureBeat is adamant about maintaining editorial objectivity. Thomson Reuters had no involvement in the content of this post.

Could information be the ultimate renewable energy source?

Perhaps. According to Jennifer Pulliam, director of innovation for electricity retailer TXU Energy, data is one of the most exciting opportunities in the smart grid — a much-needed upgrade to our power infrastructure driven by information technology.

Data, Pulliam told me, allows customers to understand how they use energy — which in turn guides them to pick the best option from a broad menu of energy efficiency and pricing options that TXU has been working to cultivate in order to stay competitive in the deregulated Texas electricity market.

Of course, making energy use data interesting and engaging is part of the challenge of selling consumers on the power of data. But opportunities in data go far beyond just what’s presented to consumers.

Data was the biggest theme at VentureBeat’s GreenBeat conference last year. Software and services that will enable smart-grid analytics “will represent one of the largest growth opportunities in the utility sector over the next few years,” according to Pike Research. The market  – with contenders from IBM to Accenture to Siemens to Itron — will balloon from a size of $356 million last year to $4.2 billion in annual revenues by 2015.

The use of data to yield energy savings is also core to many of the most promising segments of the energy efficiency market, from greening data centers to better building controls. Not coincidentally, all three sectors are expected to do well this year.

Redwood Systems and SynapSense use sensors to find energy inefficiencies in energy-hogging data centers, while JouleX offers building controls  monitoring and analytics. The latter two companies have attracted investment from GE.

Weather will be a huge part of the opportunity, too, with companies like Genscape and even Google looking to use weather data to play in energy markets. One startup, Onsemble, wants to use weather and sensors atop wind turbines to produce more accurate weather forecasts. Better accuracy in weather forecasts are then used by building controls data-crunchers and analyzing software to help users save money on their energy bills.

Building controls players like BuildingIQ, for example, loop in weather forecasts and past energy use to predict a building’s energy needs — and recommend actions to offset them, such as pre-cooling a building in the morning, when load is lighter and electricity prices cheaper, say on a day expected to get hot in the afternoon. SCI recently nabbed $15.6 million for itsbuilding predictive analytics.

Of course, it’s not just companies that can get hooked on data. Players like Motorola’s 4home, Verizon, Tendril and Intel are betting that their engaging displays will win over consumers and get them to adopt home energy and security management systems. Consumers love raw weather data too — smart thermostat maker Ecobee’s CEO Stuart Townsend told me today at the Smart Energy Summit in Austin that it’s one of users’ favorite features in home energy displays.

If data can capture energy consumers’ imaginations, then perhaps it can power much-needed change across the sector.

[Image via Flickr/LaMenta3]