Nvidia VP Automotive: "At some point we'll realize that autonomous vehicles are vastly better than human driving"
Nvidia VP Automotive: "At some point we'll realize that autonomous vehicles are vastly better than human driving"
Chip industry titan Nvidia emerged victorious at the prestigious CVPR competition. Danny Shapiro, vice president at Nvidia's automotive division, says their AI-driven end-to-end model aims to revolutionize autonomous vehicle technology.
Nvidia, a titan in the chip industry with a staggering market value of $2.78 trillion, has achieved another milestone. At the CVPR conference in Seattle last month, Nvidia secured first place in a competition to develop an end-to-end model for autonomous driving—a groundbreaking model capable of understanding and generalizing to new environments. "CVPR is an annual conference. It focuses on computer vision and pattern recognition. It's a very research-heavy conference that companies from all over the world go to," says Danny Shapiro, vice president of Nvidia's automotive division, in an interview with Calcalist.
"As part of the conference, there was a particular competition this year to do with prediction of trajectories. A traditional autonomous vehicle pipeline consists of a variety of different aspects and different neural networks looking for different things to understand the environment and to figure out the trajectory of how you would drive the car. Our research team built on a lot of work with LLMs and very large data sets along the lines of what's been happening with transformers and ChatGPT, just different types of data. So, instead of words for ChatGPT, we're talking about video from a car, but it's just data.
"The key thing here is that it's not being trained on specific environments, but on general environments. So it's not looking for something like a lane marking that it needs to attach to and steer, because if there are no lane markings the car still needs to be able to drive. This end-to-end approach from the research team was the most accurate solution out of over 400 entries in the competition."
It's the intersection of two very trendy technologies: Gen AI and autonomous driving, and you're connecting them to produce better results for autonomous driving.
"Generative AI is critical for being able to create the data sets or AR (augmented reality) data sets, and to develop the corner cases or the rare and dangerous scenarios that you don't encounter in the real world. But we can simulate the cameras, radar, and lidar signals that would reflect those types of scenes that we need to ensure that the car can handle. We both use Generative AI to create the datasets and to do the testing and validating.
"With Generative AI, we can take a particular drive and create an infinite number of permutations of that drive where children run in front of the car or things fall off of a truck, or the sunlight is blinding the cameras. The challenge is that you can't control them, so if you’re trying to drive a car and test it at sunset, where the light is shining right in the driver's eyes, you only have a few minutes, maybe a day to test that. But in simulation we can test 24 hours a day."
This solves one of the main problems of autonomous cars, that unlike humans who can develop a quick response to a new situation - they only know how to respond well to situations they have already encountered, and there is a limit to the amount of scenarios that can be generated for an autonomous system in the physical world.
"That's right. The other thing that is really key is that the simulation is repeatable. If the software needs to be updated because of, say, a child running in front of the car while the sun was setting, once we update the software, we can run that exact same scenario and see if we solved the problem or not. In the real world, you would never be able to repeat those same conditions to see if you actually fixed it."
"An autonomous car is vastly better than human driving"
Less than a decade ago, people were saying that by 2021 we’d have autonomous cars everywhere, that it would be a revolution. Then we found out that the problems are more complicated and not as easy to overcome. Is this the breakthrough or the start of one that the autonomous car industry has been waiting for?
"The easy, basic scenarios were solved some time ago, especially like driving on a highway, where everyone's driving in the same direction and there's good lane markings etc. But how do you predict what other people are going to do? That's the really hard thing. And even then being able to react even if the prediction isn't correct. If you look at what's happening, a lot of companies in China and Tesla as well with their new software, it’s more of an end-to-end model where they're feeding mass amounts of data. The advances that the Tesla software has made is quite remarkable. What they call Full-Self Driving isn’t fully autonomous but it's quite good. It’s what I drive every day, and it has gotten much better and is very impressive. There will continue to be breakthroughs from a software perspective. But the end-to-end approach is phenomenal at handling the unexpected, and being able to anticipate, based on reality and then simulation of reality.
"The issue is that there are many legal and societal aspects, perceptions of what is and isn’t allowed. Humans kill thousands of people a day on the roads. We accept that this is just a fact of life. Once there is an autonomous car, maybe there will be a fleet of cars that drive millions of miles between them, but as soon as there's one accident, that will be the biggest headline in the world, right? Even though we might have saved 99 people that day, we're not ready to accept even one fatality from an autonomous car. And so we're trying to make the software better and better. We're still seeing ways we can improve, but I think at some point we'll shift and realize that these autonomous vehicles are vastly better than the alternative, which is human driving."
Where else can we encounter generative artificial intelligence in autonomous cars?
"GenAI is a remarkable breakthrough and we will continue to see amazing results. LLMs will transform the way we interact inside the car and will allow a car that is more aware of its environment and able to communicate. For example, the car will be able to explain to the passenger what is happening while driving and its behavior, or tell the passengers it’s slowing down because a pedestrian is crossing the road, or there's a scooter that ran a red light."
Where do we stand now on the road to fully autonomous driving and what challenges still need to be resolved?
"We continue to focus on safety; that's our number one priority. There is a direct correlation between the amount of computing in the vehicle and the level of safety. We bring more computing to the vehicle to process more sensors, operate sensors with a higher resolution, and a greater diversity of sensors. We see this ongoing trend that more computation enables better software. Look at the computation capacity on the phone today compared to what it was ten years ago. We're also seeing an amazing amount of research and development on the software side that allows us to process the data more accurately and interpret the environment. The next phase is behavior: being able to anticipate what other vehicles on the road might be doing, what pedestrians might do, being able to detect not just a pedestrian, but a distracted pedestrian who is looking at the phone. Just as a human would perceive what is happening around them and try to anticipate, that’s the next phase of these models."
"People want to drive"
If you dare to bet, when will we reach the stage where autonomous vehicles will be found everywhere?
"It will happen at different times in different regions, there’s different regulation and different levels of acceptance. A lot is going on in China, where you see huge progress, development and funding. There are certain states in the U.S. where this is happening. I'm not sure it'll be ubiquitous for a long time, because we still have so many people on the road driving who want to drive themselves. The technology is mature enough today that if we took all vehicles off the road and simply replaced them with autonomous vehicles only, we would reduce the accident rate to zero, because these cars would behave in an extremely predictable way. They wouldn’t break the law or cut each other off. Ultimately, if all vehicles communicate with each other, we can get rid of traffic lights."
So the technology is mature enough for a world of autonomous vehicles, but not for a mixed world of autonomous and regular vehicles.
"This is the challenge. There are still distracted, drunk, tired drivers, you still have people who are in a hurry - who will create extreme cases. And there are still pedestrians. Wherever there is the human element, there is still a challenge. The problem is unpredictable behavior. It is not that they can't deal with pedestrians. Someone bursts into the road, there are times when a human driver isn't able to react fast enough. But a computer-controlled car will react better."
What is Nvidia's role in this ecosystem? It doesn’t make the cars, sensors or cameras for autonomous cars.
"We work with hundreds of car manufacturers, truck manufacturers, robotaxi companies, mapping companies, sensor companies, and other software companies. We developed the platform, which is the hardware that goes into the car - this is our Drive OS safety operating system. There are algorithms, there are neural networks, operating systems for the various layers of autonomous vehicle hardware, applications for drivers that we develop in conjunction with customers such as Mercedes-Benz, Jaguar and Land Rover where we’re developing the software together. We have a full set of operating systems, software and applications and customers can choose what to use. We also work with them on data collection, algorithm training and simulation."
Recently, Microsoft and Google reported that due to their investment in GenAI and data centers, their carbon emissions soared. Is this an issue that concerns you in Nvidia?
"Yes, absolutely. That's why we are very focused on how to deliver the most energy efficient platforms, and how to deliver a high level of performance without increasing energy consumption. That’s a huge part of what our engineering teams are working on."
