Hadoop, that ecosystem of open-source tools for storing and analyzing large quantities and many kinds of data, is spawning more and more companies. Some offer commercial support or consulting assistance for Hadoop. Others provide analytics software for understanding data sitting in Hadoop. But Hadoop itself is free.
Still, being free doesn’t make it perfect. Data analytics company 1010data commissioned a survey this year of 158 U.S. executives, and the results identify shortcomings of Apache Hadoop’s latest big release.
Hadoop 2.0, which became generally available in October, includes resource-management software called YARN that lets engineers run multiple kinds of Hadoop jobs on the same bunch of servers. The 2.0 edition also comes with a way to help the service survive if a key physical server in the cluster called a NameNode goes down.
Even with such upgrades, almost half of survey respondents — 45 percent — expressed concerns about reliability when asked about using Hadoop 2.0 for business projects. The cost of implementing and maintaining Hadoop 2.0 was another major issue (42 percent), along with the concern that too much coding is required (35 percent).
Hadoop distribution vendors Cloudera, MapR, and Intel have been working on security upgrades to Hadoop in recent months, but the publicly available Hadoop 2.0 on its own is apparently not yet up to snuff.
Hadoop still might not be easy to implement, either. When asked about complexity in comparison with other IT projects, 43 percent of respondents reported that it’s difficult to find people who have Hadoop experience, and 41 percent said Hadoop is new and not as stable as other available technology.
Respondents who preferred other tools to Hadoop cited simpler data integration and more affordability among their top reasons (42 percent and 35 percent, respectively).
Ask these respondents about big data as a whole, though, and almost four out of five believe it will reach its greatest potential in the next five years. Twelve percent think that’s already happened.
If these survey respondents are representative of a broader population of business executives, investors could be inclined to look for startups that make Hadoop more secure, more reliable, and less expensive.