What is HPC-as-a-service and why does it matter?

As the benefits of big data analytics becomes clear, enterprises are continuing to — or considering — adopting it throughout their organizations. Big data analytics involves examining large datasets (“big data”) generated by sources from ecommerce websites and mobile devices to social media platforms and the internet of things. A study by BARC found that, for companies which choose to embrace big data analytics, strategic decision making and operational workflows improve. Other benefits of big data analytics include better data warehouse optimization, deeper customer and social media analysis, and predictive maintenance.

But while big data analytics offers a solution to analyzing vast quantities of data, its scale requires new technologies to work with the data involved. According to Sigma Computing, 63% of employees report they can’t gather insights in the required timeframe. Most companies only analyze 12% of the data that they have. And 73.4% of companies report business adoption of big data initiatives as a top challenge.

Enter high performance compute-as-a-service (HPC-as-a-service or HPCaaS), a relatively new category of cloud service that provides the hardware, software, and expertise to process workloads including big data analytics. Cloud providers like IBM, Amazon Web Services, Microsoft Azure, Hewlett Packard Enterprise (HPE), Penguin Computing on Demand, and Google Cloud Platform run and provide access to supercomputing infrastructure in their own datacenters, usually in the form of a collection of interconnected servers that work together in parallel to solve problems.

HPC-as-a-service

Many companies with high compute requirements ran — and still run — their workloads on on-premises datacenters. In a recent Forrester survey, 44% of organizations in industries such as oil and gas, financial services, and biotech said that they currently run HPC services on-premises. But creating HPC capacity in-house can be a long process, requiring highly specialized workers and a tolerance for unexpected costs.

That’s motivating the move to the cloud, which came within reach over the past few years thanks to advances in interconnects, storage, data transport, and other components essential in HPC. According to Hyperion Research, the percentage of HPC workloads — e.g., computer-aided design, intensive graphics, and mathematical/scientific simulation or modeling — running in the public cloud rose from 10% prior to 2018 to 20% in 2019.

“HPC applications are fundamentally different than enterprise applications in their architecture, scale, and complexity. In many cases, a single HPC application spans hundreds [to] thousands of [processor] cores,” Gartner research VP Chirag Dekate told VentureBeat via email. “HPC architectures [also] necessitate dense coupling across systems — [a] specialized infrastructure stack has traditionally been challenging to leverage through cloud environments. But we are now at an inflection point where the value-capture potential from cloud-based HPC strategies will outpace traditional HPC value-capture models.”

Traditional HPC applications are adopting cloud-friendly licensing models. And HPC service providers have opened up the market by targeting low-end buyers who couldn’t afford the capital expenditure associated with traditional HPC, according to Dublin City University professor of digital business Theo Lynn, who was quoted in a recent article about HPC-as-a-service in Network World.

Hhpberlin, a German company that simulates smoke and fire damage to buildings, adopted HPC-as-a-service as it outgrew its in-house large-scale computing resources. Military contractor McCormick Stevenson Corp. similarly opted to implement a cloud HPC solution for simulating electronics on missiles and military helicopters rather than run those simulations locally.

Benefits — and drawbacks — of cloud

HPC-as-a-service offerings had other advantages. As Dekate notes, many boast capabilities that are impossible to replicate on-premises, such as dynamic or highly automated management of HPC specialized compute and storage environments. HPC-as-a-service offerings also tend to integrate newer technologies faster — Dekate estimates cloud infrastructure upgrade cycles to be two times faster than the fastest on-premises HPC upgrade cycle.

For example, HPE’s GreenLake provides a pay-per-use model for IT with options to upgrade or scale on-demand. (Infrastructure is managed by HPE, but companies can locate HPC servers at their datacenter.) Meanwhile, Lenovo’s recently announced TrueScale lets companies leverage internal infrastructure to tap into existing resources and allow HPC schedulers to access and provision resources as needed.

“Creating HPC strategies that are exclusively on-premises is increasingly unsustainable. CIOs and HPC IT leaders are grappling with the challenge of on-premises HPC environments becoming analytic islands and increasingly disconnected from broader IT and digitalization strategies,” Dekate said. “At Gartner, we are observing a fundamental shift in the market. More than 97% of [our] HPC engagements are about devising strategies to leverage hybrid cloud strategies — HPC on-premises plus cloud — and increasingly to develop cloud-native HPC capabilities.”

This isn’t to suggest that the evolving HPC-as-a-service model is flawless. Having big data analytics workloads run both in the cloud and on-premises can pose security challenges. The cost of licensing HPC in the cloud can vary depending on the type of instances used. And data, particularly the high volume of data that’s used in HPC, can’t be moved around easily in the cloud.

But Dekate believes that continued technological innovations will help to overcome these challenges. Other exprts agree — according to Allied Market Research, the HPC-as-a-service market could  generate as much as $17 billion in revenue by 2026, growing at a compound annual growth rate of 13.3% from 2019 to 2026

“The seminal use case accelerating cloud HPC adoption is the ability to integrate HPC deeper into the digital transformation journeys,” Dekate said. “Pioneers including Amazon, Google Cloud Platform, Microsoft Azure, Oracle Cloud Infrastructure, and Rescale have devised creative ways of curating differentiated HPC infrastructure stacks in the cloud.”