Physical AI Investigator
Making robotic AI more reliable through simulation, experimentation, and failure discovery
Inspiration
As AI moves from digital environments into the physical world, an important question becomes increasingly difficult to answer:
How do we know when an intelligent robot will fail—and can AI help us discover those failures before they happen in the real world?
A robot may succeed under ideal conditions but fail when friction changes, an object becomes heavier, a grasp is slightly misaligned, or contact dynamics behave unexpectedly.
Traditional demonstrations often emphasize successful actions. We wanted to build something different: a reproducible system that investigates where robotic behavior succeeds, where it breaks down, and how physical failures can be discovered systematically.
That idea inspired Physical AI Investigator, built for the Nebius × NVIDIA Global AI Hackathon.
What it does
Physical AI Investigator combines three complementary capabilities.
1. Interactive robotic simulation
I built an interactive Franka Panda robot environment using PyBullet.
Users can select bounded movement targets, observe real simulated robot motion, inspect measured end-effector position and joint states, and intervene using STOP and reset controls.
The robot uses inverse kinematics, deterministic joint control, workspace constraints, action validation, and physical feedback from the simulation engine.
This is a genuine interactive physics environment—not a prerecorded animation.
2. Autonomous physical failure investigation
Our Investigator systematically tests how physical conditions affect robotic grasping outcomes.
We created a parameterized cartridge-grasp environment that varies:
- Gripper friction
- Object mass
- Grasp-position offset
This produces 125 distinct physical configurations, evaluated using actual PyBullet simulations.
The Investigator uses an adaptive, feedback-driven experiment-selection algorithm to choose what to test next. After every experiment, it records the measured outcome and uses previous observations to guide subsequent selections.
We compared this adaptive approach against random experiment selection using equal experiment budgets.
Measured results:
| Metric | Result |
|---|---|
| Real PyBullet episodes | 525 |
| Tested physical configurations | 125 |
| Configurations producing failures | 61 |
| Average failures found by random search | 10.0 per 20 experiments |
| Average failures found by adaptive search | 14.8 per 20 experiments |
| Improvement in failures discovered | 48% |
| Selector seeds tested | 10 |
The adaptive method found more failures on all ten selector seeds.
Importantly, we also discovered a limitation: much of this improvement came from exploiting a common grasp-slip region. The adaptive selector did not outperform random search at discovering rarer failures.
Our current adaptive selector is a classical KNN-based algorithm, not an LLM. This establishes a reproducible baseline for future model-guided experimentation.
3. NVIDIA Nemotron-assisted robotic experimentation
I integrated NVIDIA Nemotron through Nebius Token Factory for AI-assisted physical scenario selection.
On October 24, 2026, two genuine Nemotron inference requests selected bounded scenarios for a simulated Franka Panda robot. These scenarios were executed using a deterministic controller in PyBullet, and measured feedback from the first mission was included in the second model request.
This demonstrates a working historical integration between NVIDIA model inference, Nebius Token Factory, and robotic simulation.
The separately released interactive Panda and adaptive Failure Investigator currently run offline. They are not Nemotron-controlled. Extending live Nemotron-guided experiment selection into the broader failure-discovery system is the next development milestone.
How we built it
This architecture separates intelligent decision-making from trusted physical execution.
Scenario or experiment selection → Input validation → Physics simulation → Physical measurement → Evidence recording → Next experiment
The main components include:
Physics and robotics: Python, PyBullet, Franka Panda simulation assets, inverse kinematics, joint-position control, contact dynamics, and rigid-body simulation.
AI integration: NVIDIA Nemotron through Nebius Token Factory, bounded model-response contracts, validation, and auditable scenario-selection records.
Autonomous experimentation: A parameterized physical experiment space, KNN-based adaptive selection, random-search baselines, fixed experiment budgets, and feedback-dependent candidate selection.
Evaluation and visualization: NumPy, Matplotlib, JSON experiment traces, physical failure-landscape visualizations, simulation replays, and a browser-based evidence viewer using Three.js.
Reproducibility: Deterministic simulation configurations, preregistered comparisons, source-integrity verification, isolated testing, and preserved experiment evidence.
The combined release passed 602 tests across separately executed validation suites.
Challenges we faced
Making simulation results reproducible. Small changes in physics configurations or experiment execution can affect comparisons. We implemented deterministic experiment definitions and validated our recorded physics outcomes.
Building reliable robot control. Interactive movement required workspace restrictions, reachable-target checks, measured joint feedback, STOP handling, reset behavior, and protection against stale commands.
Separating AI decisions from physical execution. We designed the system so that model-generated selections are constrained and validated, while deterministic controllers remain responsible for executing robot movements.
Avoiding misleading evaluation results. Our adaptive selector initially appeared substantially better than random search. Further analysis showed that the advantage mainly came from repeatedly finding an easily accessible failure region, rather than discovering difficult or rare failures. Understanding this distinction became one of our most important findings.
Managing inference and operational boundaries. We separated genuine historical Nemotron inference, offline simulation, synthetic tests, and live-provider authorization to maintain trustworthy evidence about what had actually executed.
What we learned
This project reinforced a central lesson:
Finding more failures is not necessarily the same as discovering more informative failures.
An autonomous testing system can improve a headline metric simply by exploiting known weaknesses. A stronger Physical AI Investigator needs to uncover new failure mechanisms, rare conditions, and regions where existing models are uncertain or unreliable.
We also learned how important it is to separate model decisions, physical controllers, independent safety constraints, experiment evidence, and evaluation methodology.
What's next
Our next goal is to evolve Physical AI Investigator into a genuine Nemotron-guided autonomous robotics research system.
We plan to connect live Nemotron experiment selection to the existing physics feedback loop, support bounded model-selected robot actions, and evaluate AI-guided discovery against classical adaptive and random baselines.
Future investigations will expand to rare failure regimes, out-of-distribution physical conditions, multiple interacting objects, occlusion, spatial transformations, and uncertainty-aware physical prediction.
Our longer-term vision is an AI scientist for physical systems: a platform that can propose experiments, execute simulations, measure failures, learn from evidence, and help researchers understand where intelligent robots are likely to succeed or break down.
Physical AI Investigator is a step toward making robotic intelligence not just more capable, but more measurable, reproducible, and trustworthy.
Log in or sign up for Devpost to join the conversation.