The game industry has been under attack for a long time. Security professionals have often had to deal with distributed-denial-of-service (DDoS) attacks going back years.
It seemed like the problem was solved not so long ago, but then, the vector for attacks changed. With the rise of the Internet of Things (IoT), hackers were able to get their hands on many more compromised machines, and in turn, they were able to marshal those machines in much larger DDoS attacks. And so, the game companies are finding that they are getting flooded with attacks once again.
Nokia Deepfield helps companies defend themselves against such attacks. I spoke with Craig Labovitz, general manager of Nokia Deepfield, about the game industry’s ongoing vulnerability to DDoS attacks. That may not sound like the specialty you’d expect Nokia to have, but Nokia acquired Deepfield back in 2016 to ensure real-time network security and performance.
Here’s an edited transcript of our interview. GamesBeat and Akamai will hold a breakfast at the Electronic Entertainment Expo (E3) on June 14 to talk about games and security. Contact us through deantak on Twitter if you’d like to attend.
GamesBeat: Tell us about your interest in security and game companies.
Craig Labovitz: I’ve been doing DDoS for about 20 years now. I was a founder and chief architect at Arbor Networks, one of the first commercially successful DDoS companies. I was with Arbor for 12 years. After we left Arbor, we started Deepfield about five years ago, but our history goes back 25 years doing security, doing DDoS, particularly focused on unusual traffic blocking, traffic floods, things like that.
Deepfield had its start trying to do the next generation of security for both the large cloud guys, the large game guys, and the large carriers. Deepfield was an independent company for about five years. We grew pretty quickly, to cover about 90 percent of North America. We’d just started to enter Europe and Latin America and other parts of the world when we joined Nokia, about a year ago. Since then, we’ve been able to — Nokia provided additional investment. We’ve grown our technology, grown the base. Now, we’re deployed all over the world, doing both engineering and DDoS security.
GamesBeat: Why has this problem persisted for so many years? It sounds like an almost unsolvable issue in some ways, the fact that people can still do DDoS attacks.
Labovitz: Well, I’d actually say the opposite. When we left our last company, one of the reasons I left is I thought we were done. If you go back to 2011, all the carriers deployed appliances. It’s always an arms race between attackers and defenders, whether it’s war or security. In 2011, the defenders had the upper hand. Everyone had deployed the tech they bought from Arbor Networks. Generally, while DDoS was a nuisance, it wasn’t on the front page.
Back in 2000, when we started Arbor, DDoS was on the evening news. All the major brand names were under attack. 2011, there were still attacks, but most of them were easily mitigated. Technology had advanced to a point where we thought it was basically over. We saw the market declining. There wasn’t a lot of growth. It wasn’t in the news. Everyone who was going to buy had already bought: 80 or 90 percent of the large cloud and game companies. Then, things started to change, and you get to where we are today, which of course is a very different market.
GamesBeat: 2011 was a big deal in gaming security, because it was the year of the PlayStation Network hack.
Labovitz: Right. That was when things began to change, in that time frame. I left Arbor in 2011, and in the last five or six years, we’ve seen the resurgence. As far as why things changed, a couple of things have really changed the marketplace to where you’re seeing DDoS be such a pain point for our customers and for games, as well as other verticals.
What changed is, number one, the platforms changed, in the sense of we went from compromising PCs in consumer homes to millions of mobile devices. On a regular basis, we’re seeing cloud DVRs and other home devices participating in attacks. The number of compromised devices participating in botnets has tilted the balance of DDoS back to favor the attackers.
The second thing is just the bandwidth available. In 2010, I had a megabit, a couple of megabits at home? Now, I have hundreds. Other people have gigabits. You see significant last-mile advances in bandwidth and not just to consumers. We’ve seen the explosion of cloud servers and VMs, all of which we see being used as part of DDoS today. The firepower in terms of bandwidth has grown dramatically.
Now, we’ve gone from one device in a home you can compromise to as many as 30 or 40. We’re seeing some of these IOT devices participate in DDoS — like webcams. It’s gotten much easier for criminals to hijack devices all around the world. These devices aren’t connected to just a megabit anymore. Some of them have gigabit bandwidth to the rest of the Internet.
GamesBeat: And that sends a much higher volume of junk requests?
Labovitz: Correct. The number of devices to compromise has grown by a factor of 10 or, in some cases, 100, and the bandwidth to those devices has grown in the same way. All this has really happened since 2010, 2011, where we’ve seen the balance of DDoS tilt back to the attackers.
GamesBeat: What’s been the reaction on the defensive side?
Labovitz: Well, concern. It puts you in a tough position when your attackers grow by 10 or 100 times. It’s hard to counter that. That’s why DDoS, particularly in the last few years, is making headlines again and becoming more of a challenge.
It’s a pretty fundamental shift in the way people are thinking about security. When attacks are occasional, when attacks are small, whether you’re a game company or a provider you respond by adding stuff to the network, by adding servers or different security devices. When you get to this scale of attacks, when the attackers are 10 times bigger than any capacity you have, it’s no longer a matter of just adding more devices to the network. You have to fundamentally shift how you think about security, particularly with an eye toward things like DDoS.
GamesBeat: What has that shift been like?
Labovitz: Back in the day, I used to have a Palm Pilot. I had an MP3 player. I had five different devices that I carried with me that were all sort of adjunct. Similarly, in networking, you used to have a separate device for every possible function. You had a firewall, a DDoS box, an analysis box, a router, a management box. You tried to scale by scaling up all five or six of these things, and that worked for a good 15 [to] 20 years.
The problem, of course, is your attackers are now so much bigger than you are. It’s hard to scale each of those things separately by 10 or 100 times. What you’re seeing now across the market is a shift to move away from that Palm Pilot view of the universe and look to have this embedded in the network, embedded in the infrastructure. You can’t just add it on as an afterthought.
For years, security was an afterthought. You build your network, your game, or your data center, and then, you added security to it. The real shift today is it needs to be part of how you build it from day one. It needs to be everywhere, ubiquitous, embedded. It needs to scale at the same rate you scale your game servers and your network. That’s what we’re seeing in the market today.
GamesBeat: If you had to tick off, say, five things game companies have to worry about, where would you put DDoS in that spectrum of security problems?
Labovitz: It’s kind of like asking a homeowner how they consider security. If they’ve never been burglarized, that’s the last thing on their list. Someone who’s just been broken into or someone who’s made the front page of the Wall Street Journal because they just lost five percent off their stock value, they might have a different opinion. Having done DDoS for 20 years, our best sales were the day after. We used to call them the day-after sales. The day after someone made the front page of the Financial Times, those were the easiest sales we ever had. You hear similar stories about home alarm systems.
When we started doing DDoS 20 years ago, we had to convince people they needed DDoS protection. I think the market has largely matured, and people believe they need it. The question is how much. Clearly losing all your game infrastructure for a period of hours or days is catastrophic to the business. In terms of things you worry about, that would probably be near the top of the list. Things that pose an existential threat to a company are good things to worry about.
GamesBeat: As far as where the online game operators are at, are they effectively all outsourcing this function to the likes of Akamai or Amazon? Do they say to the providers, “Hey, if I get attacked, just give me some more compute resources and get me through it?” Or, is there a different mix of infrastructure.
Labovitz: If you look at the game companies, what’s been interesting over the last three or four years is they’ve come to look a lot like network providers. They’re starting to not only do DDoS themselves, but they’re building their own data centers, laying their own dark fiber, handling more and more as performance becomes a competitive element in games. We see the top five game companies take over more and more of their own infrastructure, down to dark fiber. They’re building out their own global networks.
We did see a period of outsourcing, but now, the opposite is happening, as performance and latency and jitter become more important. As scale has grown, the major game operators — certainly in the U.S. and also in other parts of the world — have made big investments in infrastructure.
GamesBeat: We haven’t talked much about platforms yet, but are we talking about consoles or PC or even mobile? I know that on mobile now, the fast interaction has been very important for games like Clash Royale or Arena of Glory. These are multiplayer team games. They seem to be very sensitive to latency problems. If they’re getting attacked, is that another layer to the problem?
Labovitz: There are definitely attacks there. I think most of the issues we see and hear about from our game customers and carrier customers are more the first-person shooters. We see a ton of — it’s just constant. At any given time for some networks, as much as five or 10 percent of traffic is just people with Xboxes or other console games trying to block someone else.
When we talk about DDoS with respect to gaming, there are two types of attacks. One is you’re specifically targeting another consumer, trying to knock them off, knock their IP address off. The other is you might have monetary incentives. You might go after one of the main game companies and attack their servers. We see both of those. Less frequent, though they happen on a regular basis, are the attacks against servers. But we do see a constant, never stopping wave of gamers attacking each other for whatever motives.
GamesBeat: In that case, they’re going to the trouble of finding a farm to use to attack someone?
Labovitz: I don’t know if it’s a farm exactly. There are just sites that you can go to, pay $10 or whatever, and get a link. I don’t think it’s that much trouble. If you have a credit card or Bitcoin, you too can launch a DDoS.
GamesBeat: Now, we’re getting to another part of the problem, then, that something like this isn’t getting shut down.
Labovitz: No, they’re not. It used to be a big deal, to find a machine that [had] a gigabit of bandwidth. Today, you can rent one. We’ve seen an explosion of bandwidth, an explosion of devices out there, servers and others. Stuff on the edge has grown by 10 or 100 times. You’re left with the guys in the middle of the Internet facing — I remember I had a pool growing up, and sometimes, the algae in it would just explode overnight. I think that’s how a lot of game companies and carriers feel, facing 10 times the devices with 10 times the bandwidth. You can buy any of it for a few bucks.