Jay Parikh, Facebook’s infrastructure lead, may work at one of the biggest, most significant web companies of our era, but he recently told VentureBeat four stories he said apply to any hacker at any startup of any size.
“We’re all pretty familiar with what Facebook is,” he said to us in a phone conversation. “It’s a website you’re visiting every day, your memories, your Timeline.”
That adds up to a huge amount of data and a lot of heavy lifting. Every 30 minutes on Facebook, we learned, 10TB of data will be loaded into Hadoop, and 105TB of data will be scanned via Hive. More than 6 million photos are uploaded, and 160 million stories find their way into the News Feed. Every hour, we send 10 billion IMs on Facebook and check out 20 billion profile pics. Each hour, 216 billion queries run on MySQL, and 7.6 trillion cache operations will happen.
But, Parikh said, “The amount of traffic we serve, the number of users, the amount of data is somewhat immaterial; those things change wildly over time. The code changes, vendors come and go, but the focus on impact, speed, bring bold, being open…
“Those are the underpinnings of how we work and have defined us for many years.”
Those four key components of Facebook’s culture, collectively known as the “hacker way,” were important enough to make their way into the company’s official S-1 filing. And while the “hackerishness” of Facebook’s ethos is up for debate, there’s no doubt that these four principles have guided the social network to market dominance.
In our chat, Parikh shared four personal anecdotes about how the Hacker Way has been demonstrated during his time at Facebook — and how anyone can apply the same principles in their own work and life.
“If every engineer takes those four concepts and applies it to their environment and their business, they will see the benefit and have an impact on what they’re trying to do,” Parikh said.
1. Focusing on impact
Facebook puts its new engineering hires through a six-week bootcamp as part of their onboarding process. It doesn’t matter how fresh-off-the-boat or green you are — for that matter, it doesn’t matter if you’ve been an IT manager for 20 years. Everyone does bootcamp. And as part of bootcamp, you write code and push it live to Facebook’s nearly one billion users.
“On the second day of bootcamp, something I built got shipped out to several hundred million users,” Parikh told us. And as far as focusing on causatum goes, there’s nothing quite as palpable, as humbling, or as inspiring as that level of impact.
But even at tiny companies, Parikh said, shipping new engineers’ contributions in days, not weeks and certainly not months, is hugely important. It reinforce the value of the individual’s contributions and lends a sense of urgency to the work — which is quite important to the next story.
2. Moving fast
“Move fast, break things” is something of a mantra at Facebook. The idea isn’t to be as reckless as possible; it’s to remove all barriers and friction between you and your goal. Part of that, said Parikh, is building tools that keep you at the same breakneck pace even as your infrastructure and company continue to grow.
When you’re moving fast, he said, “You get to see whether what you do it working or not working immediately… You don’t lose focus or motivation as you execute.”
As an example of moving fast (and creating tools to move fast), Parikh tells us about Gatekeeper, a bit of software he calls “A/B testing on steroids.” Gatekeeper also has built-in rollout tools to get tested features live to all of Facebook’s users over a given time period. As regular Facebook users have noticed, it’s quite common for Facebook to field-test new features. In fact, said Parikh, “It’s normal for us to be running thousands of tests at a time.” And Gatekeeper was built so the team could keep running those tests and rolling out those features in parallel without waiting or pausing.
“Everyone says they move fast, but what are the metrics around that?” he asked. “As you grow, how do you make sure you’re not slowing down? Early on, you have to invest in tools that keep you fast.”
3. Being bold
“We built our first custom data center in 2010,” he said, referencing the company’s Prineville, Ore., facility, which we toured last year. “We took a pretty innovative approach, and it took us about 12 months to put it all together.”
As the company prepared to start building its next big data center, Parikh told us, “We were gonna basically take what we did in Prineville and implement the same thing. But the product teams never stop iterating and building new things, so the infrastructure teams can’t stop building, either.”
As it turns out, he said, “We literally changed everything that we thought was tried and true from Prineville. Maybe it was an insane thing to do, but we can’t stop… making bold changes.”
Those bold changes included new server confgurations, new network topology, new switches, new cables — new everything, basically.
“Most companies will bite off one of these projects at a time… we did this all in the course of a few months,” Parikh said. “The new data center took just 10 months to build.”
This principle of boldness, he continued, particularly applies to developers and others working at smaller startups and other types of companies.
“Being bold applies to everybody. You should have really big ideas and do as many of them in parallel as possible,” he said. “Create common goals everyone can work towards… but plan for faiure, and you can still overall be very successful with it.”
4. Being open
At Facebook, Parikh said, infrastructure and engineering work together as a united front, but the teamwork goes far beyond that.
“Because we’re taking these bold initiatives, we do mess up from time to time,” he said. “We don’t have a culture of blaming people or hanging people out to dry. We all succeed or fail together; we learn from each other… And all of the things in the new data center is the perfect example.”
Parikh said just about every type of Facebook engineer — software engineers, ops enineers, project managers, everyone — was working on bits and pieces of the new data center. “All of these changes were going on in parallel. It really took a unified team to execute in the timeframe that we did. If we had a fragmented team with different goals and a different reporting structure, there’s no way we could have done what we did as well as we did.”
The main part of openness that Parikh wanted to convey to startup managers and engineers, however, was the attitude of forgiveness in the face of failure.
“When people have the interest of the company in mind, if mistakes are made along the way, those shouldn’t be met with punishment. That’s an easy thing to do, do discipline that person, to have more reviews,” he said. “But those little things add up to slow the organiation down, and then the engineers get trained into being less bold.”
Parikh is delivering a talk about these four attributes at today’s O’Reilly Velocity conference in Santa Clara, Calif.