Inside Nvidia’s quest to build AI that understands physics
Rev Lebaredian, VP of Omniverse & Simulation Technology at Nvidia, told Calcalist about the vision for "world models," an AI field that allows for the accurate simulation of the laws of physics and terrain conditions.
An autonomous car is driving down the road on a cold, snowy day with limited visibility. Suddenly, a child runs onto the road. The car swerves at the last moment, avoiding an accident.
A short while later, in the same heavy snow and low visibility, another child dashes onto the road. This time, the car doesn’t react quickly enough and crashes into the child.
Now, the car is driving down the same road at sunset on a bright, sunny day. Will the glare of the sun make it difficult for the car to recognize a child suddenly running onto the road?
3 View gallery
Rev Lebaredian, VP of Omniverse & Simulation Technology at Nvidia.
(Credits: I-Hwa Cheng/Bloomberg , Nvidia)
The car’s autonomous system is designed to handle such scenarios, but how can it be trained effectively? The answer lies in "world models"—a form of artificial intelligence that enables accurate simulations of physical laws and terrain conditions. Using these models, countless scenarios can be generated and tested, allowing autonomous systems in cars and other robots to be trained under a wide range of conditions.
"They can be used to generate sensor inputs for an autonomous vehicle model—that is, to simulate driving scenarios where the inputs across all sensors remain consistent,” Rev Lebaredian, Vice President, Omniverse & Simulation Technology at Nvidia, told Calcalist. “This allows for the creation of an infinite number of driving conditions. Often, we need to test how a vehicle will behave in rare situations, such as when a child unexpectedly runs into the road. This is not something we want to test in real life. With a world model, we can simulate such scenarios and improve confidence in the AI's ability to respond appropriately. The same approach applies to autonomous drones and submarines."
World models also have applications in various industries, including transportation, manufacturing, storage, supply chain management, and pharmaceutical development.
In the pharmaceutical field, for example, world models can simulate molecular compositions of drugs and predict their effects on patients. In supply chain logistics, they enable the simulation of countless routing scenarios to identify the most efficient pathways. In manufacturing and warehousing, robotic systems using world models can perform tasks faster and more accurately than humans.
Lebaredian, who has been with Nvidia for 23 years, began his career in Hollywood’s special effects industry, working with companies like Warner Bros. and Disney. This is where his interest in simulating the physical world through computers began. "When I joined Nvidia, I started applying these technologies to real-time 3D simulations for video games," he said.
What is a robotic system?
"A robot is a system that has three core capabilities: perception—the ability to sense and interpret the world; decision-making—processing inputs to determine the best course of action; and actuation—executing actions to change the environment.
"By this definition, an autonomous car is a robot, as is a delivery drone or a machine that packages goods in a logistics warehouse. But a city or a building can also be considered a robot. A modern building has cameras and sensors that recognize individuals and grant access to restricted areas automatically. The most valuable AI systems we develop will be those that operate in the physical world.
"With machine learning, you don’t need to manually write an algorithm. Instead, you feed a supercomputer millions of labeled images—‘this is a cat,’ ‘this is a dog’—and the system generates the algorithm itself. This principle extends into the physical world. Early AI breakthroughs, such as image recognition, inspired us at Nvidia to explore how to develop AI models for real-world applications. To do this, two things are essential: powerful computing and vast amounts of data."
This approach works well when dealing with digital knowledge, such as text and images. But how do we create AI systems that operate autonomously in the physical world? One common method, used by companies like Google, is to equip autonomous cars with sensors and let them drive millions of miles to gather real-world data for training algorithms. However, this method has limitations—data collection is time-consuming, and real-world scenarios are infinite. This is where world models become essential.
"It’s impossible to develop a robust and safe robotic system without advanced simulation," explained Lebaredian. "There are classical simulations, which rely on established physical equations rather than AI. These require enormous computing power and can only provide estimated results. AI-based simulations, on the other hand, train models to learn physical laws using real-world data.
"World models are an evolution of AI simulations, extending them to entire environments. They attempt to understand the physics of the world using the same technology behind large language models (LLMs). But instead of learning linguistic rules, they learn the fundamental laws of physics. This allows us to generate capabilities similar to those of language models, but for physical environments.
"A world model that understands physics can analyze and interpret data from sensors and generate realistic predictions. For example, we can provide an image of a sunny road and instruct the model to simulate how it would look on a snowy day. Or we can ask it to create a simulation of a road trip on a rainy afternoon."
Where does the training data come from?
"The primary sources include written materials, such as textbooks containing physical equations. But the main data source for world models is video. While video doesn’t contain all the necessary information, it provides a strong foundation. The vast amount of video content available online offers enough raw data to construct basic physics models. From there, additional training can refine and expand the model’s understanding.
"Other data sources include simpler simulators, which provide information on object speed, position, and movement. These help enhance world models, allowing them to simulate realistic physics-based interactions."
