News

Nvidia's Project GR00T Promises to Transform Humanoid Robot Learning

• By John K. Waters
• 03/20/2024

In a major move aimed at advancing the development of humanoid robotics and "embodied AI," NVIDIA has launched Project GR00T (Generalist Robot 00 Technology), a new foundational model aimed at boosting the development of anthropomorphic robots in industrial use cases.

Embodied AI refers to a type of artificial intelligence that learns from interactions with a physical and simulated environment. The AI interacts with that environment via a physical agent—a robot, for example—equipped with sensors that provide input from its surroundings—things like cameras, pressure sensors, and accelerometers. GR00T is able to understand both human language and videos as input, which mean it can be fed video clips depicting human movements and learn from them.

GR00T is trained in NVIDIA GPU-accelerated simulations, which allow the humanoid embodiments to acquire skills through imitation learning from just a few human demonstrations. The process employs the NVIDIA Isaac Lab for reinforcement learning. Additionally, it can create robot movements based on video data. The GR00T model processes multimodal instructions and historical interactions, outputting the necessary actions for the robot to perform.

Nvidia CEO Jenson Huang announced the project during his keynote address this week at the GPU Technology Conference (GTC) in San Jose, Calif.

"Everything that moves will be autonomous someday," Huang told the crowd of conference attendees packed in the SAP Center in San Jose, Calif., "whether partially or fully. Breakthroughs in AI have made all kinds of robots possible, and we are working with companies around the world to build these amazing machines."

Huang also announced substantial enhancements to NVIDIA's  Isaac Robotics Platform, which includes a full suite of GPU-accelerated innovations in AI perception, manipulation, simulation, and software. Developers can use the platform to optimize for advancements in AI, the company says, including robot learning, LLMs, generative AI, and foundation models.

"Building foundation models for general humanoid robots is one of the most exciting problems to solve in AI today," Huang said. "The enabling technologies are coming together for leading roboticists around the world to take giant leaps towards artificial general robotics."

The update to the Isaac Robotics Platform brings forward new tools and enhancements, including generative AI foundation models, a robot training simulator, and the CUDA-accelerated Perception and Manipulation libraries. (The CUDA Toolkit provides a development environment for creating high-performance, GPU-accelerated applications.) Among the new tools are Isaac Lab for reinforcement learning and OSMO, a cloud-native workflow orchestration platform for scaling scale workloads across distributed environments.

Huang also unveiled the Jetson Thor, the latest expansion of NVIDIA's Jetson series. Jetson Thor is a cutting-edge system-on-module (SOM) designed to harness the power of NVIDIA's most recent Blackwell platform architecture and a transformer engine. This innovation delivers an impressive 800 teraflops (TFLOPS) of eight-bit floating-point computing power. Combined with a high-performance CPU cluster, 100GB of Ethernet bandwidth, and an integrated functional safety coprocessor, makes Jetson Thor an ideal platform for developing robots equipped with advanced embodied AI capabilities, the company says.

NVIDIA's efforts are also bolstered by collaborations with leading humanoid robot companies, indicating a broad industry commitment to integrate these advanced technologies. The partnerships aim at developing a comprehensive AI platform to support the next generation of human-centric robots, capable of transforming labor and daily life. The company is building a comprehensive AI platform for leading humanoid robot companies, such as 1X Technologies, Agility Robotics, Apptronik, Boston Dynamics, Figure AI, Fourier Intelligence, Sanctuary AI, Unitree Robotics, and XPENG Robotics, among others.

"Embodied AI will not only help address some of humanity’s biggest challenges, but also create innovations which are currently beyond our reach or imagination," said Geordie Rose, cofounder and CEO of Sanctuary AI, in a statement. "Technology this important shouldn’t be built in silos, which is why we prioritize long-term partners like NVIDIA."

"We are at an inflection point in history, with human-centric robots like Digit poised to change labor forever," said Jonathan Hurst, co-founder and chief robot officer at Agility Robotics. "Modern AI will accelerate development, paving the way for robots like Digit to help people in all aspects of daily life. We’re excited to partner with NVIDIA to invest in the computing, simulation tools, machine learning environments and other necessary infrastructure to enable the dream of robots being a part of daily life."