No matter how much you test a self-driving car, it isn’t enough. There are too many situations the artificial intelligence system has to train for and not enough testing time on the highways. So Nvidia has created Drive Constellation, a simulation platform that tests autonomous vehicles in virtual reality. Nvidia said the system will let it test the AI for self-driving cars by driving billions of miles in VR simulations.

Announced at the GTC event in San Jose, California, the technology is a fusion of two very different technologies enabled by Nvidia’s graphics processing units (GPUs): self-driving cars and VR. (We predicted this kind of system at our GamesBeat Summit 2017 event, as car software makers used games to simulate conditions for self-driving cars.)

The cloud-based system for testing autonomous vehicles using photorealistic simulation can create a safer, more scalable method for bringing self-driving cars to the roads. Speaking at the opening keynote of GTC 2018, Nvidia CEO Jensen Huang announced that the Constellation runs on two different kinds of servers.

“We at Nvidia are dedicating ourselves to solve this problem,” Huang said on stage. “It is just so important. It’s a safety problem.”

He referenced an incident last week where a pedestrian died after being struck by a self-driving car in Arizona.

The first server runs Nvidia Drive Sim software to simulate a self-driving vehicle’s sensors, such as cameras, lidar, and radar. The second contains a powerful Nvidia Drive Pegasus AI car computer that runs the complete autonomous vehicle software stack and processes the simulated data as if it were coming from the sensors of a car driving on the road.

Above: VR test driving for self-driving cars.

Image Credit: Dean Takahashi/Nvidia

“Deploying production self-driving cars requires a solution for testing and validating on billions of driving miles to achieve the safety and reliability needed for customers,” said Rob Csongor, vice president of automotive at Nvidia. “With Drive Constellation, we’ve accomplished that by combining our expertise in visual computing and datacenters. With virtual simulation, we can increase the robustness of our algorithms by testing on billions of miles of custom scenarios and rare corner cases, all in a fraction of the time and cost it would take to do so on physical roads.”

The simulation server is powered by Nvidia GPUs, each generating a stream of simulated sensor data, which feeds into the Drive Pegasus for processing.

Huang showed a demo on stage, where a human driver in VR drove a real car. That real car was outside in the parking lot, and he drove it while in VR, remotely.

“That’s like several layers of Inception,” Huang said.

Above: Nvidia expects every vehicle to be autonomous.

Image Credit: Dean Takahashi

Driving commands from Drive Pegasus are fed back to the simulator, completing the digital feedback loop. This “hardware-in-the-loop” cycle, which occurs 30 times a second, is used to validate that algorithms and software running on Pegasus are operating the simulated vehicle correctly.

The Drive Sim software generates photorealistic data streams to create a vast range of testing environments. It can simulate different weather conditions, such as rainstorms and snowstorms; blinding glare at different times of the day and limited vision at night; and all different types of road surfaces and terrain. Dangerous situations can be scripted in simulation to test the autonomous car’s ability to react, without ever putting anyone in harm’s way.

“Autonomous vehicles need to be developed with a system that covers training to testing to driving,” said Luca De Ambroggi, research and analyst director at IHS Markitk, in a statement. “Nvidia’s end-to-end platform is the right approach. Drive Constellation for virtually testing and validating will bring us a step closer to the production of self-driving cars.”

Above: Nvidia has 370 Drive partners.

Image Credit: Dean Takahashi

Drive Constellation will be available to early access partners in the third quarter of 2018.

Huang said the neural network has to be trained with precision, and that requires an end-to-end platform, from data collection by sensors to driving the car.

He said that the U.S. has 770 accidents per billion miles. But a fleet of 20 test cars can only cover a million miles per year. That’s not enough to capture all of the potential problems that can cause accidents.

Nvidia has 370 partners developing self-driving cars and components for it using Nvidia Drive.