Hear from CIOs, CTOs, and other C-level and senior execs on data and AI strategies at the Future of Work Summit this January 12, 2022. Learn more

Google today announced that its Cloud Dataproc service — a fully managed tool based on the Hadoop and Spark open source big data software — is now generally available.

The service — which supports the MapReduce engine, the Pig platform for writing programs, and the Hive data warehousing software — first became available in beta in September. And Google has enhanced the tool since then.

“While in beta, Cloud Dataproc added several important features including property tuning, VM metadata and tagging, and cluster versioning,” Google product manager James Malone wrote in a blog post.

This tool complements a separate service called Google Cloud Dataflow for batch and stream processing. The underlying technology for the service has been accepted as an Apache incubator project under the name Apache Beam.

Now that Google Cloud Dataproc is available for anyone to use — at 1 cent per hour for each vCPU — the Google Cloud Platform is a bit more equipped to take on public cloud market leader Amazon Web Services as well as Microsoft Azure and the IBM public cloud. All three already have big-data services.

Meanwhile there are also startups that provide Hadoop as a service. But the distinction for this service is that Cloud Dataproc hooks in with other Google cloud services, like Google Cloud Storage, Google Cloud Bigtable, and BigQuery.

Documentation for the service is here.


VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact. Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:
  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more
Become a member