We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 - 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!
While millions of blissfully unaware Americans grilled burgers and hot dogs over the Memorial Day weekend to celebrate the unofficial start of summer, security teams at the primary source of all that backyard barbecue fare were battling a red-hot crisis.
IT systems at JBS, the world’s largest meat processor, were showing signs of a ransomware infection. Then came the ransom demand, reportedly from a Russian ransomware-as-a-service syndicate known as REvil. Uncertain how many systems were compromised and fearing the worst, JBS officials pulled the plug on servers supporting IT and OT (operational technology) systems in the U.S., Australia, and Canada, effectively shutting down beef production across North America on the Sunday before the holiday.
The story is, by now, a familiar one. According to threat intelligence firm Group-IB, the number of ransomware attacks grew by more than 150% last year, with the average ransom demand per case more than doubling in the same period. The latest wrinkle, however, is the type of companies criminals are increasingly targeting: companies like JBS. Rather than focus on better-defended financial institutions and government agencies, ransomware gangs are turning their sights on blue-collar enterprises, the working-class companies, the makers of things.
Large, global companies built around pursuits such as manufacturing, oil and gas processing, energy distribution, and food production have some key commonalities. Most feature a blend of cutting-edge IT systems that run the business alongside more utilitarian industrial controls and operational technology that handles the machines, levers, switches, sensors, gauges, and all manner of controllers that comprise the fabric of modern industry.
Consider JBS. While slaughterhouses are an enterprise as old as time, JBS, like all modern meat producers, relies on IT and internet connectivity throughout its plants to manage myriad recording keeping and quality controls — product sorting and tracking, equipment status and temperatures, health and safety documentation. According to Beef Central, some of the challenges JBS is dealing with in the wake of the shutdown “include what happens to thousands of chilled carcasses from cattle slaughtered on Friday that have yet to be boned-out. Attempts will be made to bone those bodies out … using manual record keeping, documentation, and sortation.”
What these tech-driven industrial firms also share is an Achilles heel. Legacy OT/ICS systems, particularly those interconnected with more modern IT, can be notoriously difficult to protect from misuse. And when an attack compromises one part of an industrial firm, the fear of increased contamination and further damage often requires costly shutdowns of entire plants. The result sends shockwaves through the supply chain and the economy at large. Such companies make the perfect ransomware victims: large, well-heeled, easily exploitable, and financially motivated to get their facilities back up and running quickly.
Other OT/ICS attacks just this year include:
January 2021: Ransomware forced global paper and packaging giant WestRock of Atlanta to shut down production at several of its 300 plants and resort to manual processes to maintain the business, which serves major clients like General Motors, Home Depot, and Heinz. Two weeks into the incident, the company reported mill system production at “approximately 85,000 tons lower than plan.”
March 2021: Chicago-based MillerCoors suffered a suspected ransomware attack that left the brewing behemoth unable to access systems that control beverage production and shipments. While the outage lasted only a few days, the disruption to operations and deliveries was significant enough to warrant disclosure to the Securities and Exchange Commission.
May 2021: Colonial Pipeline, owner of 5,500 miles of pipeline carrying gasoline, diesel, and jet fuel from Texas to much of the East Coast, shut down its OT systems in response to a ransomware attack targeting its IT network. The multi-day outage crippled 29 refineries and 267 distribution terminals, and sparked price hikes and gas hoarding across the mid-Atlantic.
Pundits have been quick to assume that the broad economic disruption these incidents create signals a shift in motivation for attackers toward more state-sponsored activism. The fact is, cybercriminals remain chiefly motivated by money, something the industrial firms have and are willing to part with in the wake of an attack. That the governments which harbor such criminals may enjoy the ensuing political chaos is, for now, mostly a side effect.
Hiding in plain sight
The problems that plague security in industrial OT systems begin with a lack of visibility. The elements of industrial control systems are small, proprietary, widely dispersed, and typically not well documented or inventoried. In most organizations, OT has its own budget, its own users and aficionados, and is managed by teams separate from the larger corporate IT leadership structure.
When it comes to assessing and mitigating risk to the company at large, OT is a landscape of blind spots.
The CISO may intuitively know there are embedded systems sprinkled throughout the facilities, but naming them and describing their weaknesses is a test most would fail. When ranking security posture for OT/ICS systems at large companies, “90% or more would be poor to fair,” Ron Brash, director of cybersecurity insights at Verve Industrial Protection, told VentureBeat. “OT sites, which are often revenue generators, or the systems used for billing, reservations, inventory tracking, and so on, are severely neglected.”
Brash said OT systems security suffers from a host of technical, financial, and cultural factors such as post-acquisition consolidation, uneven budgeting for process control versus technology infrastructure, and the ever-present priority on maintaining operations at all costs.
“Businesses are literally in the business of being in the black,” Brash said. “If you have systems that can’t run without data, resources that need to be fed with work in order to generate revenue, product that can’t get to market, then it’s simple. There is no IT or OT, there’s just [degrees of] safety, reliability, and productivity.”
These shortcomings are exacerbated by what has come to be known as “IT-OT convergence,” the inexorable interconnection of ICS/OT wares with backend business IT systems. Strengthening the ties between OT and IT is undeniably beneficial for organizations looking to increase efficiency, productivity, and profitability. Knowing not just how a product is made, but where it is going, who is paying for it, and how many more will be needed next month is critical, particularly in industries such as Just In Time (JIT) manufacturing, where margins are thin and speed is essential. The melding of IT and OT systems defines digital transformation at many industrial firms, and leveraging the combined power of data, connectivity, and physical output has evolved into its own form of competitive advantage in many verticals.
“IT/OT convergence is accelerating because it unlocks business value in terms of operations efficiency, performance, and quality of services,” Yaniv Vardi, CEO of OT/ICS security vendor Claroty, told VentureBeat. “It’s good for business and it’s here to stay. But fully realizing the benefits requires mitigating the cyber risks that come along with it.”
“IT-OT convergence actually started decades ago,” Brash said, “and most organizations now are tightly coupled to these integrated systems. There are benefits, but we need to get a handle on the implications. We have to control the environment, the flows of data, and secure the systems that are crucial to operations. We need to get better at protecting them and being able to effectively recover at scale.”
Old tech, new threats
Even when all the constituent parts of an OT environment — embedded systems, I/O devices, specialized networking gear, etc. — are accounted for, the technology at work can introduce its own brand of security deficiencies. Most of what lives in an OT environment are small-but-functional computers running stripped-down versions of Linux, Windows, or some proprietary operating system.
Beyond their specialized functionality, OT wares differ from mainstream IT in the way they are treated both by the vendors that sell them and the enterprises that deploy them. It’s not unusual for OT/ICS devices to be sold with hard-coded (and simplistic) administrative passwords, for example. While the typical office laptop lasts for a couple of years and is subjected to routine security updates, OT devices can be deployed for decades without benefit of a single software patch. Even OT devices that can be patched often aren’t due to concerns about system fragility and the cost of maintenance downtime.
“There’s a 25-year gap between the state of IT and OT security,” Vardi said. “Many production environments run on legacy OT equipment that was never designed to be connected to the internet. Connecting an OT environment to the IT network means introducing an operating system that might be nearly old enough to vote, with no means of patching its vulnerabilities.”
That gap is particularly concerning for machines tasked with critical, often dangerous tasks like regulating pipeline pressures, checking machine operating temperatures, locking facility doors, or measuring contaminants in air and water supplies.
One way organizations comfort themselves when considering OT/ICS risk is with the much-overhyped “air gap.” Conceptually at least, systems with no logical connection to any other systems or the outside world should remain mostly safe from harm. The air-gapping approach ignores the possibility of insider attacks or compromises introduced by others with physical access to OT devices, of course, but the concept held up for the most part until IT-OT convergence and the emerging industrial internet of things (IIoT) became the norm. Today, true air gaps in OT are vanishingly rare. If the Iranian nuclear facility in Natanz couldn’t rely on its air-gap defense, it’s a safe bet most commercial manufacturers can’t either.
Another common strategy is to lean on what’s known as “security by obscurity.” The approach posits that arcane systems like SCADA and ICS are not well known to the majority of criminals. Reconnaissance on these systems has traditionally been difficult, and detailed descriptions of vulnerabilities and exploits typically stayed in the hands of OT/ICS specialists. This is no longer the case, however.
Over the past two years, the number of advisories issued by the Cybersecurity and Infrastructure Security Agency (CISA) describing vulnerabilities in ICS-related systems jumped more than 50%. Criminals have taken notice.
“The recent cyberattacks on both Colonial Pipeline and JBS are only a teaser of what’s to come,” Vardi said.
Strategic defense of OT/ICS
If the problem with OT security stems from it being siloed and poorly understood, the solution, experts say, is to approach risk assessment and mitigation holistically across all of the organization’s technology assets, whether they live in the office or on the factory floor. The effort begins with acknowledging the scope and idiosyncratic nature of OT systems woven throughout the business.
The rest relies heavily on security fundamentals and due diligence.
“Organizations need to practice security in breadth and security in depth to make sure that holes in the IT environment don’t allow ransomware to get into the OT networks,” Vardi said. “This includes implementing strong authentication for all OT users, segmenting their network, and ensuring complete visibility into all systems.”
Brash endorses implementing cybersecurity basics in order to reduce risk to manageable levels while simultaneously “leveraging and operationalizing the many technology investments already present within the majority of organizations.”
“Certainly for the actual ICS/OT assets this may be harder, but the majority of risk comes from the IT side. OT is generally collateral damage,” Brash added.
Organizations looking for help protecting OT/ICS and blended industrial and IT environments can turn to some purpose-built guidance and established security frameworks. Late last month, the Department of Homeland security issued a security directive specifically for pipeline owners like Colonial. The document borrows heavily from the more general NIST Cybersecurity Framework (CSF) and spells out both reporting requirements and voluntary controls designed to mitigate risk from a ransomware attack.
Just this week, the White House issued its own set of defensive best practices for private-sector organizations. The guidance calls for broader use of multi-factor authentication, endpoint detection and response (EDR) capabilities, regularly tested business continuity and disaster recovery (BCDR) protocols, and a commitment to system patching and testing.
“The U.S. government is working with countries around the world to hold ransomware actors and the countries who harbor them accountable, but we cannot fight the threat posed by ransomware alone,” said Anne Neuberger, deputy national security advisor for cyber and emerging technology. “The private sector has a distinct and key responsibility.”
For Vardi, NIST remains the gold standard for protection of all systems regardless of location or function. “The [NIST CSF] is arguably the most comprehensive and revered security framework,” he said. “Its flexibility, common lexicon, and emphasis on business drivers have fueled its adoption and recognition as a true requirement across industries globally.”
Brash said technical controls as described by NIST or in the more detailed and OT-Specific ISA/IEC 62443 standards definitely play a vital role in defending at-risk companies. He added, however, that true resilience in the face of the ransomware scourge should also include a reconsideration of processes and structure at many manufacturing and production firms in order to make systems less susceptible to disruption.
“If your worst nightmare is that you can’t schedule product to be in a pipeline, produce tracking numbers, load goods onto pallets, or get them onto trucks, then we are doing risk management, distribution resource planning, and business continuity planning wrong,” Brash said. “Ransomware is merely a symptom of the actual condition affecting most organizations.
“The good news is that we just need to rethink those processes and retrofit those organizations to get us towards the path of treatment,” he said.
VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn more about membership.