Join Transform 2021 this July 12-16. Register for the AI event of the year.

Netflix's explanation of its outage did not include the death of a network admin

(Updated 7/9/2012 with comment from Netflix engineer Ariel Tseitlin.)

Netflix has published an explanation of the Amazon EC2 outage that took down the video-streaming service for hours on June 29 and 30.

In addition to Netflix, the Amazon outage took down Instagram, Pinterest, and other sites that rely on Amazon’s cloud-based computing service. On July 4, Amazon published its own explanation of the snafu, which started with a power failure, led to a failure of the backup generators, and then caused a series of cascading failures with its load-balancing hardware.

Netflix’s analysis discovered that it, too, had problems with its own load-balancing service. “This caused unhealthy instances to fail to deregister from the load-balancer which black-holed a large amount of traffic into the unavailable zone,” Netflix engineers Greg Orzell and Ariel Tseitlin wrote in the post.

Despite the “storm” of failures, Netflix remains confident about its decision to move to cloud-based services in late 2010. The company’s resiliency to cloud failures has actually gotten better, the post argues.

“While it’s easy and common to blame the cloud for outages because it’s outside of our control, we found that our overall availability over the past several years has steadily improved,” Orzell and Tseitlin wrote.

However, the company is looking for help. If you’re an experienced cloud operations engineer, Netflix wants to see your résumé, as it is continuing to hire people for its Cloud Operations and Reliability Engineering team.

“We’re still actively beefing up our reliability engineering team…always looking for good people,” Tseitlin tweeted to VentureBeat.

Story via TNW.

Photo credit: Arthur40A via photo pin


VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact. Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:
  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more
Become a member