NVIDIA released a major open-source collection of physical AI agent skills and tools at GTC Taipei on May 31, 2026, letting AI agents directly orchestrate development workflows across robotics, autonomous vehicles, vision AI, and industrial digital twins.
What It Is
The new skills ship as part of the NVIDIA Agent Toolkit and wrap NVIDIA's existing libraries — Cosmos, Omniverse, Isaac, Metropolis, Alpamayo, and Jetson — into agent-callable tools. Instead of developers manually chaining together data generation, simulation, training, evaluation, and deployment steps, coding agents can now execute those pipelines end-to-end by following repeatable, optimized instructions.
The skills cover five domains:
- Robotics and edge AI: perception and mobility data generation, navigation training, Isaac GR00T updates, and Jetson deployment tuning
- Autonomous vehicles: fleet data reconstruction into simulation, photorealistic scenario generation, and closed-loop reinforcement learning via Alpamayo
- Vision AI: synthetic data generation, model fine-tuning, automated labeling, and live video analysis via Metropolis
- Industrial AI: CAD-to-digital-twin conversion and OpenUSD scene optimization via Omniverse
- Healthcare: hospital digital twin creation and sim-to-real policy testing
Agents run on Jetson at the edge. Security and privacy governance is handled through the NVIDIA NemoClaw blueprint and NVIDIA OpenShell runtime.
What's New vs. Prior Art
NVIDIA's individual libraries — Isaac, Omniverse, Cosmos — already existed. The shift here is wrapping them as agent-callable tools rather than developer-facing APIs. According to Rev Lebaredian, VP for physical AI simulation at NVIDIA, new end-to-end Isaac GR00T workflows can now be set up in hours rather than weeks. The Cosmos 3 model, announced alongside the toolkit, adds a mixture-of-transformers architecture that combines vision reasoning and multimodal generation — text, images, video, ambient sound, and actions — in a single open model.
Adoption and Limitations
According to NVIDIA, 1X Technologies, Agile Robots, Agility, FieldAI, Hexagon Robotics, NEURA Robotics, Skild AI, and Universal Robots are already using the agent-ready stack. Industrial partners including Foxconn, Pegatron, Siemens, TSMC, Dassault Systèmes, and Cadence are also named as users.
NVIDIA's performance claims — benchmark rankings for Cosmos 3, specific inference speed figures, and uptime improvement numbers — are stated by the company but not independently verified in the available sources. The toolkit is open source and available at github.com/NVIDIA/skills, though breadth of documentation and ease of third-party integration are not detailed in the announcement.
If the agent-orchestration layer delivers on the hours-not-weeks setup claim, it directly attacks the software integration bottleneck that has slowed physical AI deployment — a problem well-documented across the industry. That would lower the barrier for smaller robotics teams who lack the engineering depth to manually wire together simulation, training, and deployment pipelines.