The DeanBeat: My quest to defeat latency

During the pandemic, I’ve been playing games and clogging up the internet like a lot of other gamers. On the highest level, the internet has held up. Comcast reports that despite surges in demand, it has been able to keep up with our constant need to see TikTok videos, Netflix shows, and play Candy Crush Saga.

But when it comes to hardcore multiplayer games like Call of Duty: Warzone, it’s been a haphazard time. I’ve been playing the battle royale game with mixed results. I have played 443 games of Warzone, and that puts me in the top 9% of players. But I’ve only won two games and came in the top 10 a total of 68 times. This puts me in the top 21%. I’ve killed 3,079 players and been killed 4,202 times, for a 0.73 kill/death ratio. This puts me in the top 27% of players. For all the time I have spent in this game — 5 days and 7 hours — I should be better.

Naturally, I want to blame someone else besides myself. And my enemy is latency. Also known as lag. (OK, I admit I can’t really blame lag, but let’s discuss this for a while.) Latency is the time it takes a data signal to travel from one point on the internet to another point and then come back. This is measured in milliseconds (a thousandth of a second). If the lag is bad, then fast-action games don’t work well. Your frame rate can slow down to a crawl, or you can try to shoot someone and miss because, by the time you aim at a spot, the person is no longer there. So I went on a quest to figure out the problem. This happens relatively rarely in the game, but it happens.

An online game might need only 150 kilobits a second to pass data back and forth, according to nonprofit CableLabs. But the ping rate (the time it takes to reach a destination on the internet) is far more important. You can test your ping rate on sites such as Meter.net. Mine comes out to 61.3 milliseconds on a server in Dallas with Meter.net, but it was just 11 milliseconds on a server in San Jose on Speedtest.net.

Above: I’ve only won two Warzone matches so far.

Image Credit: Dean Takahashi/GamesBeat

If someone can shoot me faster than that, then I’m dead. The problem is that sometimes you get spikes in pings that disrupt your game, said CableLabs’ Barry Ferris in an interview with GamesBeat. He should know, as he and Matt Schmitt, the principal architect on the wired team at CableLabs, talked to 50 game companies about the latency problem.

Games can be designed for latency compensation, but that works only when the latency is steady. It can’t compensate for ping spikes, Ferris said. And if you try to stream data upstream, like on a Twitch livestream, at the same time as you’re playing, you’ll make the network even more congested. I record my video, but I don’t livestream because that would be way too embarrassing. But many other people do.

Is it my computer?

Above: I’m tested out lag on a Razer Blade Pro 17 gaming laptop.

Image Credit: Razer

I tried to see if my so-so performance in Warzone was due to my computer. For my desktop, I have a Falcon Northwest machine that is a year old and it has a good Nvidia GeForce 2080 Super graphics card in it. Most of my Warzone gameplay has been on this machine, which is wired into my Comcast router. I made sure it was wired because I didn’t want slower wireless results to mess up my experience.

To see if a computer made a difference, I checked out the Origin PC EVO17-S laptop as well as a laptop from Razer.

On the EVO17-S, Warzone ran at 114 frames per second on maximum settings at 1080p. It can also run Far Cry 5 at 99 frames per second, Shadow of the Tomb Raider at 94 frames per second, and Metro Exodus at 60 frames per second. It was fairly noisy, with ambient sound at 41.6dB. The EVO17-S has a 17.3-inch FHD 240Hz display. And it has an Intel Core i7-10875H CPU and Nvidia GeForce 2080 RTX Super with Max-Q and 8GB of video memory. It also had 16GB of Corsair Vengeance 2666MHz RAM. It sells for $2,941.

Meanwhile, the Razer Blade Pro 17 sports a 17.3-inch screen with a 300Hz refresh rate (or 300 times a second). It has a 4K panel driven by Nvidia GeForce RTX 2080 Super graphics. The CPU is a 10th Gen Intel Core i7-10875H processor with eight cores and a base speed of 2.3GHz and turbo boost of 5.1GHz. This machine costs $2,600 and up.

Above: The Origin PC EVO17-S laptop I’m testing out.

Image Credit: Dean Takahashi

These are the machines that should give me an edge with my ability to react to things happening on the screen. I actually won one of the two victories while playing on the Origin machine. But I encountered a tradeoff here. The fan was running so loud on the laptop, trying to cool the machine, that other players noticed. They asked what the loud sound was. I had to tell them it was the fan on my laptop. Sadly, I can’t say that the faster monitors on these laptops made any difference for me in getting victories in Warzone.

I was glad to get the second victory, but I think it was due to the fact that I was playing with some Warzone badasses at Griffin Gaming Partners: Anthony “Stembo” Palma, James “Stvrgeon” Wing, and Pierre “PierrePressr” Planche. They created this highlight reel from the game. And here’s the match from my view, including chopping folks up with helicopter blades.

I also started playing Microsoft Flight Simulator. This game streamed a lot of data from the cloud into my laptops. But it couldn’t quite keep up. At one point, I looked down while flying over the Bay Area. I saw the San Mateo Bridge stretching across the Bay. But it only went halfway, and then stopped. Because the data wasn’t streaming in fast enough, I saw the bridge stopped in the middle of the Bay, and beyond it were fuzzy details on the horizon. This is one reason why Jon Peddie Research predicted this game could spur billions of dollars in hardware spending. These laptops are very nice gaming computers, but I’m not so sure they help me win.

Getting back to the milliseconds, I’d love to talk to more people about this. But Nvidia CEO Jensen Huang said during his recent event introducing new $700-plus GeForce 3080 graphics cards, that in the game Valorant, a sniper can watch at a gap between two walls. A player character passing by that gap can cross it in 180 milliseconds, or maybe a fifth of a second. He said a typical gamer has a reaction time of 150 milliseconds. That leaves only 30 milliseconds for any other delays in the network. If you use one of the new 360 hertz displays powered by the new graphics card, you can get back 50 milliseconds and have a better chance to shoot the player going past the crack in the wall if you use the new displays.

Is it my router?

Above: Netgear’s latest Nighthawk router fits in with your gaming gear.

Image Credit: Netgear

Just yesterday, Netgear announced a Nighthawk Pro Gaming XR1000 WiFi AX5400 Router that claims to reduce lag for console and PC gamers. Netgear product manager Max Wu said it can reduce ping rates (the time it takes to send out a signal and get it back) up to 93% in a congested network.

Wu noted just how crowded our homes are getting — with an average of 15 or more devices connected to a network — and how it really gets bad when you have someone playing Warzone, another person watching Netflix, and somebody else on Zoom. If we all start using cloud gaming services like Microsoft’s xCloud, Google’s Stadia, and Nvidia GeForce Now, it will get worse. If your network is clogged already, these services will clog it even more, as it’s like sending a full Netflix stream down your pipes. Game streaming may use as much as tens of megabits of bandwidth per second, according to CableLabs, on top of needing low latency.

“This is not like other traffic we’ve carried before, as it is both latency-sensitive and high data rates too,” Schmitt said.

With the Netgear app for the router, you can tell your router not to be stupid. Because the internet was designed to function in the case of nuclear war, it sends traffic in hops to different nodes of the internet, picking up lag along the way. With the Netgear software, you can look at a dashboard map and tell your router not to seek out a router in Russia if you’re playing in the U.S. You can “geo-fence” your router so that it only seeks routers in an area that are close to you. You can also protect a gaming channel from your home to the internet, and put less priority on your sibling’s Netflix stream or other traffic.

This and other features are why Netgear believes gamers will pay $350 for this router.

“When I was a lot younger I was really into cars, and I see that people that are into PCs and competitive gaming are kind of like gearheads with cars,” Ben Acevedo, a gaming expert at Netgear, said in an interview. “Anything to get just that little extra inch, maybe it’s only one horsepower or one millisecond, you will go for it and chase those things down.”

He added, “With cars, it’s the tires that grip the road. And with online gaming, it’s your modem and your router. That’s what grips the internet. That’s what gets you where you want to go.”

Among gamers, players such as esports athletes would definitely pay any amount of money to get even a slight edge.

Is it the internet?

Above: Cox Enterprises is launching Elite Gamer.

Image Credit: Cox

Here’s a little secret. Bandwidth doesn’t fix latency. If your download speed is 100 megabits a second, your ping rate may be a certain number. If you pay for more bandwidth at 400 megabits a second, your ping rate may stay the same. That’s because cable providers tend to charge based on bandwidth, rather than latency. You could download a PC game faster with more bandwidth. But it won’t play faster. A lot of unused fiber optic cable is out there — called dark fiber — but using it isn’t going to help that much, said Schmitt at CableLabs.

But other things can. In June, Cox launched a new paid service called Elite Gamer for Cox subscribers at an extra cost of $5 a month. It’s only available in the Cox areas (6 million subscribers in 18 states), and Cox claims it can improve your response time as much as 32%. Cox provides you with software that downloads onto your machine and then identifies what you’re playing. It identifies the traffic pattern and then finds you other routes so that your game data travels the best path. It provides players with a dashboard so they can see the improvement.

WTFast does something similar, though it is not tied to Cox’s network. The company charges a fee of $8.33 a month to gamers. It finds a better way to connect a user to the game servers, and so it sets up specific connections to servers for the game. I noticed that I was getting an improvement of 10 to 60 milliseconds, or an improvement of 10% to 40%, with a different result for each game session. WTFast only works with specific games, and so you have to see if it can fix your particular game. I’m not sure I always believed it was working, but the data reports were very detailed.

WTFast actually works with Warzone, and during one moment, I felt like it made a difference. I was in Warzone in the aircraft Boneyard in the back of a plane. I started getting shot in the back by another player. I turned around and fired late. And I got the kill! The guy shouted “What the f…” before he died. I started laughing. Maybe my armor protected me and he missed. But he had the drop on me.

Above: Subspace is solving internet traffic problems for games.

Image Credit: Subspace

Another company is called Network Next. Its technology measures the lag in a particular player’s game every 10 seconds. If it finds the delay is too long, it looks for alternate paths to speed the packets over the internet to the right destination.

The company bids a price to move a user’s data at a certain speed, and the winning bidder gets the traffic. The bidders who can supply better traffic don’t know which application the user is running. The bidders only need to know which path the packets need to take and the bidders determine if they can do it at the right price. If Network Next succeeds in moving the game traffic faster than the public internet can, then the game developer pays Network Next a fee. This enables the developer to please gamers who may otherwise be unable to play because they have too much lag.

Still another company called Subspace is building out a ghost internet, or a network of private servers that can be used by multiplayer gamers to bypass the bottlenecks on the internet. The company raised $26 million in April for this purpose.

You can think of Subspace as a new kind of CDN for games. It deals with problems about why the internet, which was originally designed for redundancy in the case of a nuclear war, is screwed up. Internet packets have to hop from one kind of infrastructure, owned by one company, to another, owned by another company. Those handoffs take time, and routing isn’t as efficient as it is supposed to be.

Software alone can’t solve the problem. Part of the solution is lighting up dark fiber, or unused fiber-optic networks, and Subspace has spent part of its money doing that in hundreds of cities around the world. Hundreds of billions of dollars have been invested in submarine cable systems and terrestrial fiber networks. But companies by and large don’t have control over the global network infrastructure.

Subspace builds a map of the internet, finds the paths that are fast and routes the traffic. It’s like taking the playbook of high-frequency traders and rebuilding it for games.

Is it my cable modem?

Above: CableLabs is attacking lag for gamers.

Image Credit: Shutterstock

CableLabs recently announced that its latest software for cable modems, DOCSIS 3.1, has low-latency DOCSIS technology. This is a new approach to latency that targets a reduction in the round-trip response time to sub 5 millisecond ranges for applications. This means that webpages will load faster, video calls will be smoother, and yes, online gaming will be more responsive, based on my interview with CableLabs’ Schmitt and Ferris. Their research found that latency is what happens when data that should be moving fast gets stuck behind data that is moving slow.

As with the Netgear router, the DOCSIS software will separate the types of data you have, whether it’s video streams, online games, or other traffic. The software separates these types of data into different queues, and it prioritizes the ones that need real-time interaction, such as games. A simple software update will implement the new tech, but cable companies have to adopt it first. CableLabs saw technologies such as GeForce Now coming, and that’s why it stepped in with a solution, Ferris said.

Cable operators need to deploy equipment that supports the Low Latency, Low Loss, Scalable Throughput (L4S) technology to ensure the combination of high data rates and consistent low latency.

“We’ve been working on ways to reduce latency on DOCSIS systems for years,” Schmitt said. “We hit that point where if you really want to get additional fundamental improvements in latency, you have to look at the specific applications you’re trying to reduce latency for. Online gaming was obvious. When all of this traffic is coming into your home for a game and for video streaming, it is sharing a single queue, a single stream of data. When you have multiple kinds of traffic, that’s when you have a problem. Add to that online education or Zoom, then you have more unexpected delays.”

Above: Call of Duty: Warzone is one of Activision Blizzard’s big games.

Image Credit: Activision Blizzard

By separating the traffic into different queues, cable providers can reduce the latency. If game developers actually mark the packets as low or high priority, it becomes even easier for the cable companies to separate the traffic. That’s a lot of game developers who have to act on this issue, but fortunately, it isn’t that hard to do, Schmitt said.

CableLabs also found that they can use Wi-Fi multimedia, a specification that goes back 14 years. It hasn’t been used, but it creates tiers on a Wi-Fi network, and it can put packets into different tiers based on packet marking. This reduces the latency over the Wi-Fi network.

“We’ve been prepping the market on this, as it is part of an ecosystem,” Ferris said. “This solves a lot of the problem for online gaming.”

Schmitt said that CableLabs’ member companies, the cable companies, have to adopt all these changes to make improvements to the network for gamers. Network Next is one of the companies that has adopted the packet marking. For normal online games, this may solve a “substantial” part of the latency problem, Schmitt said.

“The good news is all of the different solutions wind up being complementary,” Ferris said. “Even Zoom has adopted packet marking. We think the latency-sensitive changes are going to create a significant improvement in the quality of experience for folks.”

Of course, cloud gaming — as well as live events and Twitch streaming — is stressing the cable network even more as it starts to get wider adoption. Right now, cloud gaming is only around 5% of gaming traffic during the pandemic, even though it’s still in the early adoption stage. L4S could help with this, but it’s a lot more challenging to manage game streaming and find what its limits are without causing packet loss, Schmitt said. CableLabs has talked with the game-streaming companies about this, as the solution is more complicated than the latency solution for normal multiplayer games.

All I can say is that when some of these changes go into effect sometime next year, then my victories in Warzone will come.