Multiplayer gaming has been a staple of video games for decades. But while internet access has grown more reliable and faster for many people around the world, multiplayer gaming is not an exact science.
Players often find their connections drop inexplicably. Companies find it hard to pull off a glitch-free launch of a brand new game that gets millions of players on its first day. Matchmaking can be frustrating as unskilled players are mixed with the skilled, resulting in a slaughter of the noobs, or new players.
And yet online gaming isn’t static. It is moving on to greater heights. Google’s Phil Harrison (a speaker at our upcoming GamesBeat Summit 2019 event in Los Angeles on April 23-24) announced Stadia, a cloud gaming platform that taps the power of graphics processors in the data center to operate a game and enable players to play that game on any device, including PCs, consoles, tablets, or smartphones. All it needs is a fast data connection to the player.
For a breakfast at the Game Developers Conference, we collected a multiplayer brain trust. It was a group of CEOs and executives who are familiar with the design and operation of multiplayer games — including some of the biggest that have ever been created.
Our panelists included Nelson Rodriguez, head of media partnerships at Akamai; Michael Condrey, head of the new game studio 2K Silicon Valley and former co-head of Sledgehammer Games; Paul Sams, consultant and former chief operating officer of Blizzard Entertainment; Anthony Castoro, former general manager at NantG, operator of the battle royale game H1Z1.
Our audience included experts like Jonathan Singer of Akamai, Richard Browne of Digital Extremes, Chris McGill of Crowdstar, Jori Pearsall of Scopely, Matthew Karch of Saber, Avni Yerli of Crytek, and others. We talked about the unsolved problems of multiplayer gaming and where it is going.
Here’s an edited transcript of our conversation. Disclosure: Akamai sponsored our breakfast at the GDC. Our reporting remains objective.
Michael Condrey: I’ve been in the industry for quite a while. 10 years at EA, at the studio that did Dead Space, and then I went over to Activision and ran Sledgehammer for 10 years on Call of Duty. I just started a studio for Take-Two in the bay area. The studio doesn’t have a proper name yet. It’s temporarily called 2K Silicon Valley. It will be renamed something much more creative soon. As a team we’re an online-connected multiplayer shooter team, from Dead Space to Bond to Call of Duty, of course. I’m looking forward to learning a lot here today.
Paul Sams: I was the long-term chief operating officer for 20 years at Blizzard. Since that time I’ve been doing a lot of consulting and advisory work in the game industry.
Anthony Castoro: I’ve been in the game industry since the early ‘90s. I did a lot of work in the online space most of my career. I worked at EA four times, although I’m not sure if one of them counts. I’ve also done some startups in the game technology space, game engines, MaxPlay. I’m currently a consultant to Patrick Soon-Shiong and acting general manager of his new game industry startup called NantG, which just acquired H1Z1 and relaunched it. We’re doing a lot of new and fun stuff in the esports and online mobile space.
GamesBeat: We kept our topic pretty broad today. It’s multiplayer. We have people represented across the entire spectrum of the industry here, from mobile to PC. I’m interested in hearing what you guys want to talk about today, rather than directing this conversation. We could start with Nelson, who might have something a little more to focus us.
Nelson Rodriguez: What we’re always curious about, and part of what drove the cloud gaming discussion last year, is the unsolved problems. All we’re ever interested in is the problems that we have yet to solve, that we can work on solving. Akamai is always thinking about big-scale problems. Not necessarily how you address 10 million concurrents, but how we can help address hundreds of millions of concurrents in [bad] network environments. That’s one of the ways we think of the problem.
For me, that’s the angle. My son plays every day. He’s in Boston. We have a one-gigabit connection, and he still gets 150ms (millisecond) ping on a lot of his games. I would say that’s not solved, even for some of the major games. Then you go to Latin America or the middle east. Some of these markets are unreachable with current infrastructure. For me, that’s the topic I’m interested in. What is still not solved in multiplayer, and what is the table doing to try to solve some of those experiences for players?
Also, how does that shape the long-term strategy? We have customers who have great big successful games, but they’re struggling to reach the middle east. They’re struggling to reach Russia. They’re struggling to reach Latin America. This is with lots of money, lots of footprint. The problem still persists. That’s what I’m curious about.
Condrey: I certainly see a lot of opportunity ahead, and some room to go and get it right. I think about the areas where an online-connected game can go wrong. There are some big buckets. Some of it is your network code, the way you write your game. Some of it’s your backend. A bunch of it is infrastructure. That can be the infrastructure in your house or the infrastructure that gets the data to your house.
When you look, from my perspective, at the places this stuff can go wrong, it’s kind of a miracle that this stuff works as it does today. The fact that you see games that have multiple millions of concurrents right now — packet loss and data centers going down and all these things that could impact your game — I think it’s a testament to where we are. It also reflects that there’s a lot of improvement to get this to be global.
Right here we’re local. We’re sitting at the heart of the infrastructure. We’re in Silicon Valley. But to your point, on a global basis, it’s a very different scenario. Thinking about what Yeti looks like, when you’re on a WiFi router in — it could be Kansas, or it could be Afghanistan. There’s some work to do to make that experience great. But there’s a lot of opportunity in that too.
Richard Browne, head of external projects, Digital Extremes: It comes back to the age-old developer thing where, well, it doesn’t happen here. Testing it is another thing. Knowing this, being able to deploy this. We launched a game in December. It came out at the Game Awards. 5,000 people flooded onto the server and the thing just fell apart. We didn’t have a robust enough testing system to get 5,000 or 10,000 people into the game at any one time. And that was in a game we launched in the states. We need some sort of infrastructure or company to step forward and say, “We can test that for you, anywhere in the world, whenever you want.” That’s a service somebody could sell an awful lot of.
GamesBeat: EA set up this whole case study for itself by launching two games in a month. They got 50 million users in a month for Apex Legends and then Anthem had a fair amount of complaints when it launched two weeks later, even though they had a fairly staged rollout, from the people who paid the most to the people who paid the least.
Rodriguez: Over the last year, Akamai has acquired and developed a technology similar to that. Testing at scale, synthetic testing, is really tricky. Getting it to feel like a real thing is hard. Making sure you’re testing on all of the data centers against all the endpoints, that’s difficult. We’ve found that to be an exciting opportunity for us.
Browne: And it’s not a single use case. What I find fascinating is that I have two children, 14 and 11. Watching the way they interact with games now — for us, when we were growing up, you were just there. You sat in front of the console and did whatever. Then it went online, so I’m playing with my friends online, and I’ve got broadband to my apartment and it’s all great.
Now I watch my kids play, and they’re doing nine things at once. It fascinates me, because I can’t understand when kids learn patience. [laughs] With Fortnite, you go into it, you get on the bus, you land, you run around, you get headshot, you’re back to the lobby. This loop — kids 10 years ago would just turn the game off. But now they’re watching YouTube in the meantime to see how to play the game better. You have Facetime running over here. It’s not even an issue with a single connection coming in like it used to be. It’s the fact that there are 20 people in the house. Having those test cases, with all the things that can go wrong, that’s going to be very important.
GamesBeat: Paul, do you see this across multiple companies?
Sams: Quite frankly, the answer differs based on what access to capital you have. When I was at Blizzard we had tremendous access to capital, and we were able to put our own infrastructure out there. We weren’t relying on anybody else. We built an entire network and we set up all the direct colocation with all the major ISPs around the world. We had a gigantic team and infrastructure to be able to manage that. We were running multiple GNOCs and doing a 24/7, follow the sun kind of approach.
In all candor, we were able to just spend and figure it out. We had a strong user base that was supportive, and much more tolerant of some of the challenges we faced in the earlier days. Now, dealing with smaller companies, where they don’t have the same access to capital as we did, it’s a much more challenging endeavor. I agree. The need to test and make sure things are comfortable and bulletproof, that they really work well, is a real challenge. There’s a huge opportunity.
What you’re going to go face, you’re going to have a lot of access to capital as well. With that and all the experience you have, it will probably be a bit easier than what some people at the table might experience. It’s definitely a place where you guys might have something to apply.
Condrey: It’s a curious case study you’re talking about around the two games that launched for EA. This is from an uninformed perspective, but one might project that the core difference between those games is strictly game readiness. One game is ready and one game is not. Everything downstream, what it takes to have an online community, doesn’t really matter, because the game’s just not ready.
Castoro: And in that case it seemed like the consumer was aware of that. That was fascinating to watch. One of the biggest challenges for multiplayer experiences — I’m working with a joint venture now between Big Rig and the new entity of NantG. We took over H1Z1 and relaunched it. We’re going through a lot of — hey, on the internet there’s no guarantee as far as quality of service from the user to you. You can build out a lot of private infrastructure, but you still have an issue with that last mile, that last 100 feet or whatever.
It really isn’t about bandwidth. You have a kid in the house or in my case four kids in the house, who are all watching their own shows on Netflix or YouTube or whatever. They’re chatting with their friends and playing video games. It’s not bandwidth. Most of the time the issue is latency, and the quality of service, the consistency they have. For mid-sized and small developers, that’s a real challenge. You don’t have the ability to basically set up your own private network on the internet for people to connect to.
And even if you did, you don’t have control over that router that hasn’t updated in the last three years or rebooted in the last three months. That represents a significant challenge. The amount of testing you do doesn’t provide a developer solutions for those kinds of problems. It’s truly an internet and infrastructure problem. Even 5G may not solve that problem.
GamesBeat: Isn’t that the thing Google didn’t talk about? They can control 5G and bring that into the home, and then they can have an all-Google network that will just be up to them to manage. I think that’s what they want.
Audience: What’s the reason that carriers haven’t played a bigger role in the solution here? Carriers, a lot of times, will come to us and say they want to build a gaming-specific solution. You have the middle mile. You have maybe a private backbone. You have the carriers who control down to the user. But there’s always been a struggle, it seems, to get code down the client that can interface with something specific that carriers are doing. What’s the reason that’s been the case?
Castoro: I’d speculate that part of it is just that they’re always trying to outdo the others. They always want something unique to them from an operating standpoint, which is hard for the developer to support. If each ISP I use has some kind of unique integration? That’s one possibility.
Anjos: Developers could also take some of the onus on themselves. The more casual a game is, the bigger the chance that they’re using a wi-fi connection. What providers don’t tell the paying consumer is that you have a higher chance of packet loss if you have a wi-fi connection. Developers can take an extra step forward to really have a breadcrumb for the player. “Here are the steps you can take to have a wired connection. With a wired connection, your gameplay will be more stable.” That comes from a developer that’s really pro-play experience. It comes from production or producers on that side, to have the consumer in mind and really feel their pain.
Audience: Thinking about it from a mobile perspective, you bring up an interesting scenario. I would be afraid to try to do that on mobile. If I say, “Don’t play this on LTE because your experience is going to be poor,” I’m going to lose that customer. That balance ends up between the perfect experience and getting nothing. Hearing some of this stuff on wi-fi and wired connections — when you talk about mobile, we do a lot of analysis, or try to, around, say, what is the experience of a player on a train? Where you have no guarantee of everything. It’s hard to keep a good user experience in that situation.
Anjos: David Baszucki at Roblox is putting every single penny they have in profit into infrastructure engineering. Their high-level view is to have a scenario where a player in Korea playing on Android can have the same experience as someone in Chile on a high-end PC. No matter the connection. They’re trying, and they’re putting a lot of money into it. I can’t wait to see what they come up within the next few years, because they want 4 billion people concurrently.
Rodriguez: I don’t think that’s even biologically possible. [laughs]
Condrey: That’s one of the challenges I think about a lot. Last I checked, nobody has been able to change physics. This idea that we’re going to have 8K, 60 frames per second streamed content to devices that are largely wi-fi connected — right now that feels like a fantasy. Would that be great? Yeah. Is it today? No way.
Understanding how we get to a place where the experience is right for the in-house network is interesting. What we saw — most homes now have high-end console boxes that use wi-fi. That right there, the laws of physics say that all these big things about whether or not a streamed backend can process and render your game back to your client really isn’t — you can have the biggest data center in the world. But to your point, if you have a misconfigured router and every kid streaming Netflix at the same time, the laws of physics are still in place.
Castoro: I was just having those OnLive flashbacks.
Audience: I also want to caution everyone who’s getting a little too bullish on 5G and what it’s really going to do. Having come, before Akamai, from specifically a 4G technology standards body, I’ve lived in that world. What you’re seeing right now is carriers saying that they have 5G coming. What it really is is LTE Advanced, which is actually the ITU standard for 4G. They’re just marketing it.
The actual 5G, that’s coming slowly. It’s going to be really expensive for them to roll out. They’re only going to roll it out in major markets. It’s only going to help people who already have great connections. They don’t know how they’re going to price it, who they’re going to charge for it, how they’re going to charge for it. They’re going to be looking for more money because it’s a massive investment.
There’s excitement around 5G. I certainly talked it up a couple of years ago. But I was telling people to get prepared for a seven or eight-year timeline for that to really matter. Not next year, not in two years.
Condrey: There’s a really particular thing that can help solve for the — it may not apply to everyone, but one of the things that might be beneficial early on for things like 5G is crowds with mobile phones. At Disneyland or at a football game, that experience is so bad right now. I don’t know if you’ve been to Disneyland or Disneyworld recently, but it’s mostly people walking around playing Pokemon Go until they can get on a ride. If a bunch of people are on a raid and they’re taking up the network connection, 5G can help with that kind of multiplayer experience earlier than others.
GamesBeat: I’m curious about how you can design yourself into maximum trouble. Netmarble’s Lineage 2: Revolution, or Blade & Soul: Revolution, those have 100-player PvP battles on mobile, or possibly thousands. That sounds like they’re designing it for South Korean networks, but not ours.
Condrey: Battle royale games on mobile, same thing. If you’re trying to get hundreds of people, or even tens of people, into a shooter in a mobile environment — we’re seeing a lot of people trying that stuff right now.
I wonder about the applications for AI and machine learning in testing these things. It seems like there could be some relevance to figuring out how to get better testing in a distributed environment using AI, just because leveraging lots of people is always a big challenge in an international operation.
GamesBeat: I did have a chance to sit down with Phil Harrison yesterday, and I asked him about this longer-term trend of — everyone I hear talking about computing now says there’s way too much data. We’re going to have the internet of things. Masayoshi Son talks about a trillion connected devices. These are the things that will contend for the network with games.
They’re all talking about pushing intelligence to the edge of the network. AI processing, if it’s going to reduce the flow of data into the data centers, which is important — you do the processing at the edge. You get all that data from a sensor, you analyze it, and then you send what’s important up to the data center. That means we’re going to have smarter devices at the edge, which is the opposite of cloud gaming.
Rodriguez: Well, it doesn’t have to be. Cloud gaming, the way we think of it now, maybe looks at super PoP infrastructure and data centers that are sitting clustered. But if you look at — Jonathan at 2K was talking about micro-services, and particularly things you can do at the edge — databases at the edge, key value store at the edge — which would allow for things like saving state at the edge. That means you don’t have to go all the way across the network in order do something like — what we call cloud gaming could be edge gaming, depending on how it’s architected.
It’s not a simple problem to solve, to have unified state that can dynamically go from edge to edge. But it’s doable. They don’t have to be mutually exclusive.
GamesBeat: It still sounds like there’s a lot of contention for the network. 5G fills up just like that.
Rodriguez: Oh, sure. There are different parts of it. When we’re talking about your house or your building and how many people are on the network at once, that’s one challenge. But if we’re talking about trying to get from San Jose to Seattle and back, parts of that can be solved with the edge. It won’t solve in the house. It won’t solve the last 10 feet. But it can solve a lot of the last 10 miles.
GamesBeat: Phil’s answer, by the way, was that Google will and wants to control most of its network, as much as possible.
Castoro: Again, it’s a quality of service thing. Should your refrigerator’s processing of how old your food is prioritized as much as your PUBG session? But then you have a net neutrality conversation. Refrigerators are important. [laughter]
Rodriguez: I’m curious about people’s thoughts on the overlap between cloud gaming and multiplayer. If someone is playing on a device — if you’re basically playing on a server, is multiplayer going to be easier, because the servers are just connected to each other? All you’re worried about is the one player’s connection. But if they’re playing with 1,000 people and all of them are connected in one data center, is that a simpler problem?
Benjamin Charbit, CEO of Darewise: We’ve been talking with a company called Shadow about trying to store the game engine closer to the backend. Currently it doesn’t seem to be such a massive problem, but you still have that last mile issue. Until we can figure out how to handle that, I’m still quite skeptical about how this is going to massively change things.
GamesBeat: Has anybody else tried Shadow? I’ve tried it. I played Apex Legends on it. I tried to go through as many matches as I could until I could get a kill, and I couldn’t get a kill. That might be me, but then I went back to the PC and I started getting kills again.
Audience: Could you feel that, or was it just the stats?
GamesBeat: Between pushing the inputs and having something happen the way I wanted it to happen, it felt like there was something slightly off.
Rodriguez: What’s the opposite of the placebo effect? [laughs]
Audience: We have a co-worker who’s tried it. She was playing Overwatch for a couple of months, and she was saying that she didn’t see a noticeable difference. But I know, having looked into Shadow for a bit, it has a lot to do with your configuration, your speed, where you are, what your cables even potentially look like.
Charbit: There’s an interesting direction here, which is to use degradation. What you’re always trying to do, obviously, is deliver the best and most polished experience. It’s obviously something we’re trying to do well, because we’re trying to do a very large-scale open world. Even if we figure out how to handle the backend side, we still have to figure out how we’re going to show you that 1,000 people or 10,000 people shooting at each other. You still have that bottleneck on the client side.
One direction, obviously, which might be acceptable is what Netflix has been doing since the beginning of their streaming service. They’re degrading the quality so that you can still have a good experience. You can have a smooth experience by adjusting the visual quality. This is what Shadow is trying to do, trying to dynamically reduce the quality. That could help. I have the feeling that at some point, though — people are really looking for high-end graphics all the time.
Condrey: It depends on your audience. I’m guessing that the more casual your audience, the more sensitive they are to the quality, whereas competitive gamers, the first thing they do is turn all of the settings to low, because they want to win. Which drives me crazy when I watch them on Twitch.
GamesBeat: Shadow is also saying they’re going to do one GPU in the cloud per user. They can have an RTX per user. But they’re going to charge $35 per month for that. That’s the equation that they’ve figured out will work. I’m very interested to hear about what Google is going to charge, how they’re going to make money, because that seems like a physics problem again. A physics problem combined with a business problem.
Audience: And what’s the publisher’s angle on that business model? Do they get anything? Obviously there’s free-to-play. Free-to-play has changed the economic dynamics. But still, if you’re working for a triple-A studio that’s publishing $60 games, how interested are you to be on a $20-per-month service?
Sams: If I had on my old Blizzard hat, I would tell you that they want to own their entire player base. They haven’t ever chosen to go and partner with anybody else. They’ve done everything on Battle.net, short of being on the consoles. They had to make that choice, but otherwise they’ve been hesitant. They want to be able to capture all of the revenue and all the profit, without having to share. That’s their mindset.
Browne: It’s a very interesting equation. Everything has changed so much. One of the things that I railed about endlessly at THQ was the day that I had put an agreement on Darksiders II–how do you stop the game from churning? The shift from being able to do single-player games — that went away. You look at Dead Space. An awesome game, but five years earlier, Dead Space probably would have sold three times as many copies. You had people buying it, keeping it for 12 hours, and selling it back to the store for credit.
There’s this interesting parallel now where the streaming services are like — well, I’m going to launch my game at full price. And then instead of losing any money on the back end, because Gamestop’s just going to buy 1.5 million copies of Darksiders II and churn it, now you have a secondary market where my tail is back again. It’ll be on a streaming service and I’ll get income from that. It may just be a little royalty check every now and again, but it’s better than nothing, which is what we had at a certain point.
At the same time, you’ve got somebody who’s subscribing to this Netflix service where there’s now so much content that — why would you go and buy $60 games at all? I don’t know what your Steam backlog is like, but I have so many games on Steam that I’ve never touched, because I just bought them in a bundle or whatever. There is still a call for that, but from a publishing standpoint, like you say, I think the publishers are still going to want to own the audience. Ubisoft wants to own it. EA wants to own it. Activision wants to own it. Maybe they’ll put it on the Google cloud 12 months down the line.
Audience: Not coming from console, there’s an open question — in mobile you’ve seen this transition over the last few years of live ops, live ops, live ops. On the cloud services, does that come over more and more heavily to triple-A titles? Releasing new content every month turns your game from something everyone buys the first week of launch to something that can keep growing over a long time.
Browne: Warframe is a great example of a game that’s now six and a half, seven years old. It started as 15-man team and basically took over Digital Extremes. It’s become extraordinarily successful. It flew under the radar a bit, but it makes an awful lot of money considering it’s made by an indie and it updates all the time. It’s almost a weekly conversation that grows and grows and grows. That’s the only way to do games live.
When I’m talking to external developers now, I’m talking about completely different ways of thinking about building games than ever before. When I talk to them I say, “When you pitch this IP to me, are you thinking out five or 10 years?” Not 18 months. Do you really want to stay with this game for that long? What’s the one-hour experience, the one-day experience, one week, one month, one year, five years? Why am I still playing this game in five years’ time? That’s the only way to approach it now.
I don’t think we’re fighting for dollars anymore. We’re fighting for time. Warframe is a hobby. Anybody who gets into Warframe — I play Warframe a little bit. The user experience when you play Warframe is really fun. When you get to that first build-out screen for your Warframe, you’re either terrified by it and run away, because you don’t have enough time in your life, or you say, “That’s the most amazing challenge. I’m going to get totally into this. I’m going to spend months building out my Warframe, and then I’m going to get another one and do the same with that.”
You have people who come to TennoCon with their founders’ T-shirts every year and they’re revered as gods, because the game exists because they helped fund it in the early days. It’s getting that community and that fan base that’s never going to leave your game. You have people playing Call of Duty who are just going to play Call of Duty, because that’s their brand and that’s what they play and that’s what they know. Trying to get them to leave that and go to something else is extremely difficult.
Audience: You’ve brought up some good points in terms of both the mobile model, as well as scaring players away. Every time a game gets an update, content or otherwise, it’s like a 1UP mushroom in Mario. It gives the game life. It brings new attention while you prep the next launch, the next title update.
For Warframe, what you said about getting scared as a new player coming in, that lends itself to — the developer needs to really have a clear experience, a scaled experience for the player. But the trick is not to have a regret from existing players, people have been there since day one, who may feel like they’ve been burned by the developer. The only game in the triple-A space that I can think of that really does that well, like a mobile game, is Fortnite. There are items available right now that will be gone next week.
Audience: What happens to the free-to-play era? How does that business model work in an age of cloud gaming?
Rodriguez: If you look at some of the video versions of this, like the OTT market, there’s a lot of OTT that’s ad-driven. In those cases it’s a revenue share model. I don’t think it would be unreasonable, if there were a free-to-play game that wanted to be on a cloud streaming platform — I would imagine that the cloud platform is going to take a cut of transactions. It’s how it works today on most video services.
Browne: Apex Legends, to me, is a very interesting test. It’ll be fascinating to see where it stars in EA’s financials the next time they announce. Or it’s probably not the next one, but the one after. It’s either going to be the headline of EA’s earnings report, or somewhere in between. It’s great that they have 50 million people, but are they monetizing? I think they have a great game. They’re on the right track with it. But they have some issues. Fortnite, you have one character, you add stuff to that one character, and so all they have to do is keep adding stuff for people to apply to those characters
But to your point about playing in the cloud, if they take 30 percent or whatever — to me, it’s our business. It’s what we’re trying to do as developers. For us it’s just about removing any barrier to playing the game. We want somebody to be able to come in and just try it. There are lots of really great games on Steam and the PlayStation store that don’t get the attention they deserve because there’s $20 worth of barrier. Back in the old days at Psygnosis we had game demos on the cover of CGW, so everyone played this Lemmings demo. The good old days.
Free-to-play is a lot like shareware as a concept. You’re giving something to somebody and — the way I try to frame it, and I think Warframe does a good job of this, is trying to build games that aren’t necessarily — they’re free-to-play, but you want to pay. What I mean by that is, the consumer plays and has so much fun with it that they find a way to give you money. That’s what we’re looking for. I think Apex has done a good job of that. I know a lot of people have bought the extra two characters because they want to keep playing with them. But I’ve also heard from a lot of the same people that they’re not so interested in buying more characters because they’re happy with what they’ve got.
That’s why it’s such a different mindset, building a free-to-play game. From the ground up, thinking about the way a game works is totally different than the $60 experience. Free-to-play could become ubiquitous if it wasn’t for quarters. At the end of the day, for Activision and EA and Ubisoft — and this is why I’m interested in their earnings report — to get away from that Red Dead billion-dollar weekend launch is hard. It’s really hard to invest in something different.
GamesBeat: Everyone was very critical for EA about how many different launch slots they had for players in Anthem. But I guess the good thing about that was they were trying to stage the ramp-up for Anthem.
Audience: One of our marketing partners — they haven’t actually released it. But when EA put out the Anthem advert that said, “If you’re a member of Xbox Live you can play it here,” and they had that whole spreadsheet of release dates for players — they had the Javelin going here like this. And so our marketing partner had the Warframe doing this and it said, “If you want to play Warframe, go ahead!”
GamesBeat: If I think of a game that was designed to be friendly to the multiplayer network, to be playable, I think of Hunt: Showdown. It’s just two players teamed up with each other against a few other teams. Crytek designed a fun game around that smaller experience. It seems like even though you have these gunslinger duels that happen really fast, you’re not trying to put 1,000 people in one match. Is that deliberate thinking?
Avni Yerli: It’s a new concept for us, games as a service. Showdown was designed so that Crytek could be transformed quite easily into a service-oriented company. We learned a lot. We made a lot of mistakes. But the idea of the design wasn’t to challenge too much with the game size. We’ve been able to learn a lot in that environment. It’s worked well for us. We had to play to the strengths of the team. We didn’t want to come up with design decisions that would create too much trouble. Somebody said earlier, if you lose a player it’s very difficult to get them back. You need to make sure that they have a good experience and pass that experience on to other players.
GamesBeat: As far as other things you can control, I wonder about the game developer or publisher’s point of view on things you should do before launch, at launch, and after launch. Do you have tips on that front?
Anjos: Before launch, you want as much data as possible. Data is the language that customers use to speak to us. If there’s an issue in the data that we can narrow down and fix before launch, that’s invaluable. Any problem that a player sees in the beta, that’s something 100,000 players might face during launch week, and that’s a PR disaster. I would stress using that kind of analytics early on, because that itself has its own launch in terms of data flow, data pipeline, and what you need to instrument and implement and prioritize as the game’s launching.
GamesBeat: Call of Duty was doing betas, what, four or five months before launch? It’s interesting that Anthem had its beta only one month before launch. You can’t change the game very much one month before launch.
Condrey: I think of two different ways that betas inform the game. Some of it is qualitative. You’re looking to see how players react to the game design. Part of it is readiness. If you’re trying to figure out your readiness at scale in your beta, that’s a problem.
Call of Duty had the luxury of time and big teams. We had built a lot of tools around simulation of the experience. We had a bot farm that ran matches all the time, every day, constantly. We had tools that would input lag into our internal network to simulate what that would look like in the real world. Betas are important, but the challenge — I haven’t made a battle royale game yet. This was a six-on-six experience. Just having 12 people playing all the time is a limiting factor.
In our private betas we rolled out to 5,000. In public betas we’d roll out to several hundred thousand. On day one you’d have millions of people showing up. The scalability of your game at launch is really hard to simulate prior to launch. That’s where a lot of things tip over. To the extent that you can build tools around what this is going to look like when it goes live — that’s what we learned along the way. We could never fully test what was going to happen in the live environment, but as much as you can do to simulate it before then — again, on Call of Duty we had our own problems too. But for Anthem I feel like that was maybe a misstep.
Castoro: I’m going to back that up, especially the tools part. Even smaller organizations, and maybe more so smaller organizations, should be investing in tools to try to automate and test. There are so many resources out there, going back to AI and machine learning applications for those things.
The other thing I’ll suggest that often gets overlooked is security. Whether you’re small or large, the environment out there now for multiplayer games, especially with the cultural differences between, say, China and the west, presents really difficult challenges for developers and operators of these games. I got some WeChat just earlier this week about players who discovered that there’s some entity out there paying other players to cheat in my game. They’re all speculating about who’s backing that, whether it’s a big publisher or a rival or whatever.
We know, and that happens all the time, but finding players or services or people out there who will hack your game for you — being armed with that information is extremely helpful, especially if you’re on a boutique engine versus something like Unreal. You have different considerations there. But if you’re making a multiplayer service, you really need to think about how you’re going to handle hacking and cheating if you’re popular, because it’s inevitable.
Condrey: We would get DDoS attacked six times a day in Call of Duty. It’s a good point. There are steps you can take before you launch to make sure you’re ready.
Castoro: There are multiple levels of it. There are service attacks, like a DDoS attack, and then there are just hackers and people moving around in your game, creating items, things like that. There are so many different fronts that you have to worry about for multiplayer services.
Anjos: Also, we’re in a world with GDPR compliance. Every country has their implementation of that, and how data has to be stored. That also has to be in the conversation as the backend services are being set up to make sure you don’t get sued by Lithuania, say, or some other country. Not many small developers are even considering that. They just want to make a great game.
Rodriguez: How do you handle the security discussion? It’s going to depend, of course, on the size of your company, but we’ve just talked about three different points of security. There’s attack resilience. There’s the fidelity of your actual gameplay. There’s the fidelity of your in-game economics. There’s account security. There are all these different pieces of it. And there’s obviously the pre-launch phase and then when it’s actually live and happening to you. Then you figure out where you really need to focus, because that’s where the pain is. But how do you make those decisions ahead of time?
Castoro: If you’re doing a multiplayer service, it has to be maybe second only to your user experience considerations. It’s a user experience issue. If someone is trying to play your game and they can’t reach it because you’ve been DDoS’d, or they can’t win because some guy is teleporting all the objects in the game to himself, including the players, and lighting them all on fire, they might think that’s funny, but they may not keep playing for very long.
It has to be in the DNA of your company. Back in the day we would do things like just hiring hackers. Now there’s a whole ecosystem of services around that, from third-party anti-cheat solutions to internet infrastructure solutions around DDoS and how you automate changes. It just has to be native to what you do if you’re making a multiplayer game.
Condrey: And it’s an unsolved problem, really. Especially in the PC space, which is an open platform. You can ban. You can have a bunch of data collection devices that can, after the fact, identify cheaters and try to change the culture. But it’s an unsolved problem.
Castoro: Steam has come up in my mind twice in this conversation. One, they just announced an update to their API where developers can start using their infrastructure, their network, which I thought was a really smart thing on their part, because it is such a challenge for smaller developers. It presents an interesting opportunity, depending on how much they’re investing in that space and in security.
The other side of that with Steam is that they do a lot of the account handling. Depending on how your service works, sometimes you can rely on a third party to deal with some of those things. But as you grow, the different service offerings, the different publishers and operators and storefronts, present a whole bunch of ecosystems for multiplayer developers to consider. I was happy to see Steam make that move.
GamesBeat: I heard them talk about it. They mentioned that they built the network because of Counter-Strike, because Counter-Strike requires a 30 to 60 millisecond response time. If they have that, they can let everyone else use it.
Castoro: The other thing they did, and I think it’s mostly because of CS:GO, is their anti-cheat, which they talked about last year. They use machine learning to identify potential cheaters and surface that to their panel of players and decide who gets banned or not. Their ability to accurately identify cheaters was greatly improved by using machine learning to look at play patterns, look at where cursors are moving and how far they’re traversing and in what direction.
Maybe they’ll be releasing those things to developers as well. Some of those are very specific to that type of game. But if you look at what they’ve had to do with their game, that shows some of the challenges that multiplayer PC developers have to deal with in general. You see how much they had to invest to deal with that.
Audience: That’s something that’s also very hard to simulate. In the conversations I’ve had about anti-bot or anti-cheat measures, collecting all the human telemetry about mouse movements, things like that — yes, you see that in real time, but in a test environment, how do you simulate a lot of those events? That’s been a challenge, at least when I talk to a lot of game developers.
The question I always ask — from a security perspective you’re looking at account checking issues, or DDoS issues, but what are your anti-bot measures? It almost seems like not even a secondary issue, but a tertiary thing that they look into. I’d be curious to know where that falls into this whole security discussion, once you go past DDoS. DDoS is almost the basic level.
Castoro: If you’re a competitive game, or if you have a skin market or something like that where players can make money from your game, you’re going to have a huge problem with bots and automated cheating. On Z1 Battle Royale, which used to be H1Z1, with Daybreak and their MMO background, we have all kinds of security things that we’re always working on. We have things that will take pictures of the desktop of someone we think is cheating. We have systems for looking at those to automatically recognize that a certain part of the UI shouldn’t be like that. But a lot of those things happen after the cheating, not during the cheating. The Chinese cheat program people put out a new update every day. They adapt. It’s an escalating war.
A lot of it can ultimately just come down to social engineering. If we think you’re cheating, we’re going to put you in an arena with other people we think you’re cheating with. We’re going to try to figure out which players are which by how they’re behaving and what buckets they fall into. It’s a nasty job. It’s very difficult.
Audience: We’ve talked about this with some of our customers, like EA. I’ve played a lot of Madden Ultimate Team, where virtual currency abuse is crazy. All the YouTube influencers are directing people to rockbottomcoins.com, and it seems like — they don’t really care. They see it as self-promotion. They’re generating their own brand from the game. That’s a huge problem.
Castoro: This is a massive issue, and it doesn’t get talked about in the industry very much, maybe because it is such a big issue. Because there’s so much money at stake in those aftermarkets, the depth to which the hacking will go, even with some of these bigger companies, is almost unimaginable.
One of the ways to deal with it is to try to not acknowledge it, so you don’t make it official and you’re not liable for it. That’s the approach that a lot of companies take. “We don’t endorse it. We know it happens, but if we really acknowledge it, then it becomes a problem we have to try to solve.” Instead, you try to focus on your direct consumer experience. Only when it affects those people do you have to tackle it. But it does affect their behavior. They’re motivated to try to make money, and that changes everything.
Anjos: PR is more important than ever. We saw that with Fallout 76. The scale of the problem took them off guard. They didn’t have a robust PR system to maintain messaging that was concise, consistent, and frequent. Any overture they made, they had to reel it back in, and that just caused even more negative consumer sentiment. That’s what I saw as their biggest issue at launch, despite all the other issues out there. The messaging kept confusing players. You had different sources of information that were officially from Bethesda, and they were all backtracking.
GamesBeat: That whole topic of managing multiplayer communities is a fun one. There are a lot of toxic folks out there. I get insulted on a daily basis when I play with other people.
Castoro: I run a game that has free and open proximity chat for everyone who plays. I must be insane. That’s a whole other topic for multiplayer games. Those are quality of service issues, things like chat. The voice chat in Apex Legends was fascinating to see, how poor the quality was and how much consumers seemed to notice it.
Audience: That raises a question for me. You were talking about how companies of a certain size want to control everything themselves. But you see the rise of Discord, and the idea that some services can be provided by third parties that you don’t have any control over, whether it’s chat services or messaging or third-party marketplaces. To what extent, understanding that there’s maybe a top-heavy bias in our conversation — to what extent does it make sense to rely on third parties for some of your core services?
Sams: It’s absolutely necessary. Even at Blizzard they don’t want to deal with — they don’t want to control that. They know they’re not going to be able to stay in front of that particular tech, a company that’s dedicated to it. If it’s not part of the gameplay, then that type of organization will consider it.
With voice chat, Blizzard used a third party solution. They weren’t trying to build it out themselves, because even though they knew it was important for the experience for the players, they didn’t feel like it was an area where they’d be able to deliver a result that was superior to what a third party could do, and they didn’t feel that it was so core to the experience that they had to do it themselves.
There are just certain parts of the overall set of touch points that are critical to managing control. They also use middleware in a variety of ways, because they don’t need to do certain things. It’s just not crucial for them.
GamesBeat: Do you see everyone coming out and embracing things like Discord? Consumers are enthusiastic, but I wonder about publishers and developers too.
Castoro: It’s one of those interesting things. You want to understand your players, understand how they like to operate. At the same time, it’s something that’s not necessarily in your control. You can’t control the quality of the experience. Integrations like those are — you have to think about what you’re good at, what your consumer wants, and what your overall strategy is.
Friends lists are another good one. A lot of us have our own services where we have accounts with our own friends list, but how we integrate the Steam friends list, the Epic friends list, or whatever storefront you’re on? What about Discord and how can connect things up so players aren’t having to manage 17 different social networks while they’re trying to just play your game? It’s an interesting challenge.
Audience: As far as what developers may not value as much as consumers, even now, even to this day, I don’t think developers know how much in-game representation matters to their players. In Call of Duty, for example, you had a variety of racial demographics. We actually matched the usage of certain characters, certain races, to the 2011 U.S. Census. It was one to one. People in this racial demographic would pick this version of a character. You couldn’t tell the difference.
That’s an opportunity, I think, to have more overtures from developers. You can’t even see your character in a first-person shooter, but people would pick that anyway. Imagine how much impact it can have to actually see a character who represents you.
Castoro: That was interesting with Apex. I remember the screen coming up, looking at all the characters, and thinking, “Where’s the regular white guy? Is he the guy in the mask?” As someone who’s not just a regular white guy, it was interesting to see that choice come out of that product.
Audience: I’m curious about the future of $60 multiplayer games.
Castoro: Aren’t we all? [laughs]
Anjos: I’m going to quote Strauss Zelnick in a recent interview about how to justify that kind of price. He said something very insightful. The average cohort that plays their games is 37 years old. As more players play Fortnite, they’re going to get old one day. They’re going to want that experience. I don’t think there’s any threat to that $60 packaged goods game if there’s something that players can really invest in.
Charbit: If you can monetize your product over the long term, where’s the threat? It’s just a different distribution of the revenue. We’re all starting to come around on this. We did it on Assassin’s Creed. We’ve started bringing different kinds monetization into the experience. I don’t see it as a threat. It’s just an evolution of the business model.
Audience: There are three models we can see. There’s free-to-play, straight up. There’s $60 and charge me later, which has obviously become an issue. And then there’s $60 and play forever. We’ve mentioned Wall Street’s response to going purely free-to-play. I think consumers don’t care as much. I think consumers care about $60-plus IP. But as far as $60 versus free-to-play — obviously consumers are excited about free-to-play. That’s something they’re comfortable with. I’m not sure if publishers are.
Audience: There’s also a different expectation of experience. If I pay $60 it had better be a Call of Duty-quality, visually stunning game. If I’m a free-to-play game, my route to market is very different. I tend to see it like $60 games are movies and free-to-play games are TV series. You’re building a much lower entry point. There are test points as you go on — does the game play all right? — and then there’s the retention thing. How can I retain players? If I give them a fun experience and I have a content pipeline that keeps giving them stuff, they’ll stay with and they’ll pay. They’ll end up in a position where they want to pay.
I think the more interesting thing, again, from watching kids growing up, is this whole construct that you have to build God of Wars and Red Dead Redemption 2s for hundreds of millions of dollars. We love that stuff. We love that visual fidelity. But Fortnite comes on and it’s simple and stylized. My daughter has more fun playing that than she does playing God of War. It’s all about the community aspect of it, having fun with her friends, playing it with everybody. It’s what they’re talking about at school.
The construct of going and playing a $60 game that’s a single solid experience is something that resonates around this table a lot, but I wonder how long that goes on, and how everything does just shift over to community and multiplayer. Then the business model is really about expectations on entry. I’m going to pay $60 to play a game that looks awesome, the visuals are fantastic, and it’s going to be a solid game from the day I pick it up. Then I pick up something that we might build, on a much lower budget, and I think, “There will be issues with this.”
That’s why everything we do is community-driven. It’s getting people in early, getting those founders in, and getting them to hopefully support the game, because we don’t have the financial backing to have hundreds of people in customer service answering questions all the time. We expect the community to do that. Warframe today is still crowdsourced from a localization standpoint. We have people around the world saying, “We really want this in Czech!” Well, we can’t do it. Then you get three or four people doing it themselves.
It’s a different way of approaching it. I don’t think either is a better way, or that one is going to make the other extinct. To my earlier point, it’s a blocking point for huge companies to say, “We’re not going to deliver that.” The talk of Fortnite killing Take-Two’s stock price — people are going to start buying Take-Two stock as soon as the release date of GTA or Red Dead comes out, and then the day that Red Dead launches and gets its billion dollars, I’ll sell it, because I know that’s the last thing from them for a while. [laughs]
GamesBeat: There are interesting dynamics with free-to-play when it hits its end of life. You see games starting to lose their audience. They move on to something else. The developers and publishers start to put the squeeze on their remaining players, because they want to keep the revenue stream going. If they want to continue to reinvest in the game, they have to have that revenue stream, but if it keeps going down, then you just try to milk more out of the players that are left. There were some very organized boycots in mobile games.
Kevin Chou from Forte pointed this out, and suggested that maybe blockchain will bring some help to those kinds of economies, giving more variety of things to monetize, like unique collectible items. But his observation was that free-to-play is broken in that way, when you see it over a longer period of time. Somebody has to figure out how to better monetize the 98 percent of players who aren’t paying for anything.
Castoro: Part of that is that the business model isn’t fixed. You don’t have to stay that way.
Audience: Crowdstar is a little different. Our approach is — it doesn’t wear out, because it’s a game, but it relates to your life outside the game. Covet Fashion is in its sixth year, having a record year. Most games want to solve the problem of, “I’m bored.” We’re trying to entertain you, but we’re also trying to give you value that you can relate to outside of the game. We’re constantly introducing real things into the game that you can play with. In theory, that doesn’t wear out. It’s a different approach.
Audience: When we were building these games, on a personal level, I had been thinking that free-to-play was broken for a very long time. The monetization mechanic of, “I’m gonna give one audience a shitty experience, but you can get the good experience if you pay,” it just doesn’t seem right. You’re punishing most of your audience to squeeze money out of that one percent.
I’ve thought about this over the years. Why not create monetization mechanics that are more like pay-to-play than free-to-play? It’s still free-to-play, but at some point, if I want to keep playing for another 30 minutes today, that’s where I start investing money. Not spending $100,000 to get the biggest and best character. Machine Zone is a great example. Their strategy was to put out a sale, and then the very next day, they’d kill that economy. You paid a bunch of money and then you’d get a really bad experience.
Audience: The Warframe approach to that problem has always been — your game has to be able to hold up. If you asked anybody on the Warframe team if, in five years’ time, the game was going to be an open world where you could fly into space and jump across spaceships and have combat in space, they would have thought you were nuts. But the IP always had that ability to grow and grow.
The problem you’re speaking to is that a game just stagnates. When a game stagnates it’s always going to start dying. Only so many people are going to want to play it, and then they filter out and your retention goes down. If you build a game that’s always changing — when we do TennoCon every year there’s always something new announced. There are 2,500 people who fly to London, Ontario to say, “Whoa!” Even though they could just live stream it.
That’s the obligation of the team. There’s a reason 300 people are in London, Ontario working on the game. It has to evolve, month on month and year on year. There’s a new challenge, a new Warframe coming out, something new to do. We have that core community. It’s all about that community.
Audience: What would you say is the best example of a game that’s evolved like that?
Anjos: GTA Online is a five-year-old game. It was number one on Twitch last weekend because they ran a 30 to 40 percent sale on in-game items. Every single update for that model is very unique. You can pay in-game money, or buy that money, to create business in the game. You can make more money on that in-game business for more in-game stuff. Roblox is a step further, where you had 70 million monthly active users last month. 40 million players in Roblox develop their own games inside Roblox.
Audience: But Roblox isn’t a game. It’s a platform.
Anjos: Exactly. But it’s free-to-play. You can engage with other people’s games and pay in-game currency that creators get a cut of. We’re going to get to a point where — free-to-play is today. In five years I’m going to play a game not because it’s more fun, or because it’s cheaper, but because I can make money from it. Fortnite is trying to get on that as well with supported creators.
Audience: That’s a limited addressable market, though. We address a 35- to 40-year-old woman. She’s not going to do that. I would argue that Roblox is not a game. It’s a platform that other people build games on. That’s how it evolves, through other people building temporary stuff on it.
GamesBeat: I did have some interesting conversations this week based on all the Google announcements. The Shinra Technologies folks became GenVid. They felt like Google announced what they were trying to do years ago with cloud gaming. But if there’s a different kind of game that results, something you can create related to this, involving streamers playing games and the audience of the streamer piling into this and changing the game, participating in the play in some way, then you’ve created a different kind of game that’s ready for this world of streaming.
Castoro: Participation gaming, where you’re not playing a game, but now you are. Or building the game as a player.
GamesBeat: Mixer was designed for this, with a low latency to enable this to happen.
Audience: We were talking about $60 versus free-to-play. How and when do you make the decision that your one game is going to become someone’s hobby? If you look at Warframe, that’s a hobby. People play that specifically. For some people, Call of Duty is their game. Do you have to do that when you conceptualize it? But then you have to react to player feedback. How does that change things?
Audience: Testing. Early testing. We talked about third party solutions a bit ago. The more third party solutions you put in your game, the quicker you can focus on your core gameplay and get that tested in the hands of users — don’t wait until two months before launch to show the game to 100,000 people.
Audience: Even beyond that, we test value propositions before making them. Accepted consumer belief, stated belief, we put that into testing. You understand what she’s doing, how she’s living her life, what problems she has, and what she enjoys. Then you start, and then you test.
Castoro: The question you’re asking, though — as you get to actually launching, how do you know if what you’re launching is going to meet the thesis? We used to call it the big book of stupid. You’d come up with the design bible of what the game is, and then it’s never that when you actually ship it. Steve Gray just announced that he has a company that’s all about predictive analytics, having all this data and learning from Tencent and the way China runs all these free-to-play games.
GamesBeat: He said that within the game, they’re testing so many things that are happening. Not just testing the reaction to the game, but every single thing the player does.
Castoro: That’s one of the tough things for game developers to figure out when they’re developing a game. Everything they do needs to have telemetry, needs to be trackable, needs to be something you can ask a question about later. When you’re developing a multiplayer game, you’re not thinking about that. Doing those things ahead of time, making it part of the game, and using third-party solutions, especially for the small organizations, is the only way to do that.