A firmware rollout I was running had been moving zone by zone for six days when it stopped itself. The page always comes as a silent vibration first, but at 1:14 AM it feels like a fire alarm. In one zone, the share of hosts that rebooted uncleanly during the update had crossed the safety threshold, so the automation froze the rollout and flagged it for a person.
The first assumption in a moment like that is that we broke something. We hadn't. The hosts that failed had been stuck for hours before the update reached them, on a hardware hang we already knew about. The reboot in the update just made it visible. We changed the check to leave out hosts that were already unhealthy before the update, resumed, and finished the rollout 9 days later.
That incident taught me something I still apply: A health signal has to be measured against the host's state before the change. Otherwise it blames the release for damage that was already there.
None of this involved a bad chip design, a bad model, or a code bug in the usual sense. It happened in a layer most people don't think about when they buy AI capacity: The release process that keeps firmware, drivers, and run-times moving across tens of thousands of machines while jobs keep running on them.
I've come to think that layer now sets the limit on how much useful work an AI cluster delivers. Hardware at this scale fails constantly. Most of the fixes and mitigations for those failures ship as firmware and software. A fix for a known hang helps only once it's on every affected host, and getting it there quickly without breaking anything else is the release process' job.
Hardware fails on a schedule
Meta's paper on training Llama 3 is unusually open about this. Over a 54-day stretch of pre-training on about 16,000 H100 GPUs, the team recorded 466 job interruptions. 419 of them were unexpected, roughly one every three hours, and about 78% of those were traced to confirmed or suspected hardware issues. Meta also reported that synchronized training made the data center's power draw swing by tens of megawatts.
A small per-part failure rate, spread across tens of thousands of accelerators, a very large number of high-speed links, and racks that draw more than 100 kilowatts each, turns into a steady stream of incidents. That isn't a failure of engineering. It's arithmetic. Multiply a small per-component failure rate across tens of thousands of accelerators, vast numbers of high-speed links, and racks that swing from idle to peak power in milliseconds, and "rare" becomes routine.
The metric that captures the cost is goodput: The share of paid-for compute that actually moves a model forward. Google Cloud uses the term for exactly this. Rough arithmetic: If a job fails every three hours and checkpoints every 30 minutes, each failure throws away about 15 minutes of work on average. Add about 15 minutes to detect the fault, swap the node and restart, and the job loses half an hour every three hours, about a sixth of its time.
Fixes ship as releases
When I started in release engineering, a release meant pushing new code to servers that mostly stayed the same underneath. Today the stack under one training job has multiple independently versioned components: Accelerator firmware, management controller firmware, network card software, drivers, run-times and diagnostics, and others. Each moves on its own cadence, and all of them have to work together on every machine in a job. A typical release on my team changes most of them at once.
Deploying to a custom AI accelerator fleet is a total rethinking of the data center, not a card swap: The cluster behaves like one disaggregated supercomputer, with chips linked by dense point-to-point fabrics that make signal integrity a high-stakes priority, racks drawing well over 100 kilowatts that make liquid cooling the norm, and thousands of ASICs leaping from idle to peak in milliseconds.
Because chip, interconnect, rack, cooling, and compiler are co-designed as one system, a fault in any layer surfaces everywhere — a marginal cable masquerades as a software hang, a power dip looks like a failing chip — so floor operations adapt with rigorous burn-in stress testing, orchestrated maintenance coordinated with cluster schedulers, and specialized mechanical handling for the dense chassis.
That makes release engineering partly a hardware discipline. Three things work differently.
Most bugs have to be caught before the canary. We still roll out in waves, starting with a small pilot zone, and the incident above shows the waves doing their job. But production health checks catch loud failures, like hosts that don't come back or instances that won't launch. Many of the failures that matter only show up when hundreds of chips communicate under full load, or after hours of sustained work. A pilot rack can pass cleanly and miss them. So every release first runs for a few days on dedicated test fleets, under collective-communication stress tests, memory tests and sustained training benchmarks.
Rollback gets tested as hard as roll-forward. In ordinary software, rollback is usually one click. Here, some updates apply live and some need a reboot, some change state that can't simply be undone, and an older version of one component may not work with a newer version of another already on the host. Our test fleets move hosts forward to the new release and back 20 times over four days, and a release isn't eligible until both directions are clean.
The rollout is a scheduling problem. A fleet-wide release goes out over 2 weeks, region by region. No two zones in the same region update on the same day, the largest zones go last, and the plan works around holiday freezes and other releases still in flight. Health signals can stop it at any point, as they did in the incident above.
Silent bugs that passed every component test
Here are a few examples that passed component-level testing. Each of these changes was correct on its own. Each one broke in combination with something else.
The power cap that slowed training: A firmware update lowered the default power limit per accelerator so that a full rack would stay inside its power budget. Every component test passed. In system testing, the step time of a standard training benchmark got 5% worse and failed its performance gate. The firmware team had expected the slowdown and accepted it. But nobody told the people who owned the performance gates, and the earlier per-component tests had thresholds loose enough to miss it. We shipped the previous version, which cost 3 days. The bug was in coordination: An intended tradeoff never reached the team that owned the gates.
The buffer that overflowed on the biggest hosts: A host management agent collected inventory from each accelerator card into a fixed-size buffer and never cleared it between cards. A new version made each card's data about three times larger. On hosts with the maximum number of cards, the buffer overflowed on the last card, the agent lost its view of the whole host, and no instances could launch on it. The cards themselves were fine. It reproduced on only one host configuration, which is why system testing found it and code review didn't.
The common thread is that the failure was never inside the change. It sat between the change and something else: A power budget, the largest hardware configuration, the rest of the chip. The only way to catch that class of bug is to test the whole system the way customers will run it.
A failure that stops a job is, in a sense, the easy case. The ones infrastructure teams worry about more are quieter. One is the straggler: A machine that isn't broken, just slightly slow. Because training synchronizes every step, one slow node holds back every other chip, and it never trips an alarm because nothing is technically down. Hunting stragglers is a recurring theme in the large-scale training papers.
Catching these means reading low-level telemetry continuously — link error rates, memory error counts, thermal and power patterns — and pulling hardware before it fails, not after.
Where AI agents help run the release
AI is beginning to help run AI. All of this testing produces more signal than people can read. Over the past year, my team has started using AI coding agents in release operations. We don't use them to run the fleet on their own. They do three narrower jobs.
Triage: Release testing produces a steady stream of failure tickets. Agents now read them, cluster them, and in one recent case collapsed ~40 tickets into 7 root causes, several of them the same hardware issue reported many times.
Cleanup: When a test run is abandoned, it can leave dozens of hosts stuck partway through a rebuild across several systems: A provisioning request in one, a hardware-vetting workflow in another, a reservation still held in a third. An agent can find every leftover piece and propose the exact order to undo them. The order matters, because some systems refuse to cancel while another one still holds the host.
Drafting: Release summaries, change records and the "is this failure a blocker?" justification now start as agent drafts that a person checks and edits.
The limits are just as real. Every action that touches the fleet still needs a human decision, and it should. Agents get things confidently wrong, for example by reading the wrong field, or by "verifying" a change against an output that never contained the evidence.
Recently, an agent told me all 10 hosts from an abandoned test run were back in the pool and confirmed it by quoting the output of the cancel command. That output only showed the cancel requests had been accepted; when I checked the hosts themselves, three were still mid-rebuild. The leverage comes from humans reviewing faster, not from humans stepping away.
What this means for infrastructure teams
If you run AI infrastructure, a few things follow from this:
Measure goodput per job, and track how long a fix takes to go from merged to running on every affected host. That second number is the one most teams don't have.
Measure health against a baseline. A rollout check that doesn't know what a host looked like before the change will stop good releases and miss bad ones.
Budget test fleets as part of the cluster. A release that's only been tested on a few hosts hasn't really been tested.
Staff release engineering like a hardware function. The people doing it need to read firmware logs and link telemetry as well as pipelines.
Start agents on triage and cleanup, and keep people on decisions that touch the fleet.
Racks keep getting denser, and every hardware generation adds components that ship on their own schedules. Chip makers set the ceiling on performance. How close a cluster gets to it depends on how safely and quickly its operators can change it, every week.
Vinayak Vashishtha is an infrastructure and release engineering leader.
Welcome to the VentureBeat community!
Our guest posting program is where technical experts share insights and provide neutral, non-vested deep dives on AI, data infrastructure, cybersecurity and other cutting-edge technologies shaping the future of enterprise.
Read more from our guest post program — and check out our guidelines if you’re interested in contributing an article of your own!
