Every time a car automatically hits its brakes because its driver failed to notice an obstruction, thank queasy computer gamers.
The technology underpinning advanced driver assistance systems (ADAS) and autonomous cars got its start solving a much more pedestrian problem – giving gamers fast-moving images of three-dimensional (3D) environments without making them vomit.
In 1992, id Software launched Wolfenstein 3D, a game that let players sneak through a German castle in World War II and fight Nazi soldiers. The game was an instant success, but many players struggled with the first-person view of the 3D environment – getting nauseated almost immediately.
The problem was frame rates – computers couldn’t render 3D environments fast enough for the human eye. Even the best processors of the early 1990s weren’t up to the task, resulting in jerky, stuttering images. Dozens of companies formed to develop new ways for computers to create virtual, 3D environments. Nvidia, one of the most successful of those startups and a company that has since become a major supplier to the automotive industry, formed in 1993.
Graphics processing unit (GPU) chips that began popping up in the late 1990s took virtual 3D environments and translated them onto two-dimensional (2D) computer screens (see sidebar, right). ADAS and autonomous driving systems face the opposite problem – taking 2D images from cameras and other sensors to create a 3D computer model that tells cars what obstacles surround them.
“We essentially have to recreate the 3D world around us as we drive. So, sensors, the cameras, the radar, and the lidar help us build an understanding of that environment,” says Daniel Shapiro, senior director of automotive at Nvidia. “Inside the brain of the car, it starts to look like a 3D video game. It’s the same type of math from video games taking place inside the car. Except now, it’s mission critical.”
ADAS and autonomous systems use 3D data as well, such as radar and laser scanning, but the bulk of data comes in two dimensions.
Nvidia chips began appearing in automotive infotainment systems more than a decade ago. Any system with a screen benefits from graphics acceleration, so those early automotive systems were basic image accelerators, not the more-sophisticated systems used today to manage ADAS.
“Early on, our customers taught us how to be automotive grade,” Shaprio says. “The design processes, manufacturing processes, the quality assurances processes are all proven in the automotive industry.”
Early this year, the chip maker announced plans to focus more on autonomous and ADAS technology, saying its automotive growth would come from advanced computing, not colorful touch screens. In July, the company partnered with German automotive supplier Bosch and Mercedes-Benz owner Daimler to develop a self-driving taxi fleet with testing set for California next year.
While the Nvidia Drive systems the company provides to automakers are designed for harsh environments – servers running in data centers don’t experience the heat and vibration the typical car does – the underlying chip structure is nearly identical to the ones that power Nintendo’s Switch game console.
Teaching cars to drive
The convergence of video game graphics processing and autonomous driving isn’t limited to the chips; software tools designed to simulate spectacular car crashes when vehicles hit each other in games are also teaching self-driving cars to avoid such collisions.
Thomas Heermann, senior director of automotive and conceptual design for design and simulation software provider Autodesk, says video games advanced simulation technology as programmers sought ways to better recreate real-world environments. The blocky cars from early racing games such as 1982’s Pole Position evolved into near-perfect replicas of real-world vehicles for 1997’s Gran Turismo (see sidebar page 18).
“Companies developing these autonomous technologies have to train the systems to react to real-world problems – a child running into the street to chase a ball,” Heermann says. “It’s a huge application for the technology. They can test scenarios quickly, change variables, and test again to see how systems react.”
Shapiro adds that video-game derived graphics technologies are getting more sophisticated every year – modeling how shadows change depending on cloud cover and time of day. Bizarre scenarios – such as light glinting off a window into a car’s ADAS camera, creating a confused image – can be tested because video game designers wanted to create more immersive experiences for players.
“We can then put those simulations through our Nvidia Drive system to see how it responds in real time,” Shapiro says. “We can simulate 60,000 miles of driving in one hour. That gives us the ability to drive every road in the U.S. in two days.”
Using GPUs to power safety systems is the most direct way video games have influenced advanced vehicles, but the industries have other ties – one of the biggest being they both use many of the same software.
Computer-aided design (CAD) systems predate modern 3D video games by decades, so action-oriented games did not lead directly to industrial design systems. However, gaming technology has greatly influenced and advanced the tools that designers use to imagine vehicles and every component used in them, Heermann says.
“Making visual decisions is super important. Designers want to have as much realism in the decision making as possible. Tools like VRED, our visualization tool, are pushing the limits on representation, especially in real time,” Heermann adds.
Decades ago, CAD data could only be depicted as a series of geometric shapes – useful for determining specifications and basic characteristics, but not great for evaluating the look or feel of a product. Shading, coloring, and texturing tools developed for computer games and animated movies create human skin that looks like skin and clothes that move in the breeze. Motor vehicle designers are using those tools to more fully simulate how a vehicle will look at different times of day or in different weather conditions.
“You’ll have a complete car with lots of variation,” Heermann explains. “The goal is to look like the real car on the street, with the ray tracing (for lighting effects), with ambient shadows on it. That’s very similar to what you’re seeing from games’ engines.”
While clay models are still common, many design studios first create vehicle concepts in digital environments. Some are even using virtual reality (VR) headsets to walk around the 3D models and view them from different angles. In late July, Autodesk announced a partnership with Prague, Czech Republic-based VRgineers Inc. to make Autodesk VRED data viewable on VR headsets. Heermann says multiple designers in studios scattered around the world will be able to examine and alter the same 3D model in a virtual space, collaborating in real time – much in the same way Fortnite and Call of Duty players team up in games.
“The same interactivity from games, designers want to have that in the studio. Realism is very important,” Heermann says. “The big difference is speed. On the game, you’re optimizing your representation for speed. No one’s really judging the car in a game for its perfection. The auto guys want to see everything in as much detail as possible.”
Because of increased competition, automakers are releasing more versions of vehicles, faster than ever before. Historically, it could take as many as seven years to go from initial sketch to a vehicle on the road. Today, launches can happen in as few as three years. CAD tools are responsible for much of that increase, but Heermann says better visualization and design tools, coupled with the ability to collaborate across continents, are playing a big role in getting products to market faster.
Coming full circle
To a large degree, the video game industry’s impact on the automotive world is payback. The visual 3D rendering tools that id Software engineers used more than 25 years ago for Wolfenstein 3D owe a lot to CAD. Software companies developed CAD to help automotive engineers and other industrial designers better visualize physical objects in a digital world, and game developers adapted those wire-frame tools to create entire worlds.
It’s possible that many visualization technologies would have developed without research and development from video game hardware and software companies, but that movement of the technology from the industrial world to the consumer market and back again has certainly accelerated development.
So, next time someone complains about video games rotting players’ brains or that violent games contribute nothing to society, think about cars avoiding accidents – all because a handful of gamers got sick to their stomachs 30 years ago.
IMTS 2018 Booth #133222