Facebook today is revealing a new feature of its data center facility in Prineville, Oregon: a cluster of roughly 2,000 smartphones on which it tests out its applications to make sure they keep working as they should.
Just a year ago, each Facebook software engineer tested out each code change on just one nearby device. The trouble is, not everyone has one particular mobile device — people have many different kinds of devices, with different operating systems and varying network connections, so this testing system wasn’t ideal. Hence, the 60 new, custom-built racks in Prineville, each of which holds 32 phones.
Facebook runs its CT-Scan program for every new version of a given application on all of the devices, with the help of Mac Minis for iPhones and Open Compute Project Leopard servers for Android phones, as Facebook production engineer Antoine Reversat explained in a blog post.
For apps that run at scale, like Facebook, Instagram, Messenger, or WhatsApp, every possible device configuration is relevant, because hundreds or thousands of people could be using it, and if there’s a problem, lots of users could get turned off and switch to alternatives. So thorough testing becomes extremely important.
Market-leading cloud infrastructure provider Amazon Web Services (AWS) offers independent developers a test bed of mobile devices through the AWS Device Farm, and the Google Cloud Platform offers Firebase Test Lab, although it’s currently only available for Android. But rather than rely on a public cloud to run its applications, Facebook has its own infrastructure, so it’s not surprising to see Facebook assemble its own armada of devices for testing purposes.
Facebook is keen to double the density of the racks for its phone test lab, Reversat wrote.
He added that Facebook is looking to share the designs of the racks for this hardware, presumably through the Open Compute Project the company established in 2011 and has since used to circulate designs for its compute, storage, and networking gear.