Inspiration

Robotic control policies often pass clean nominal tests in simulation, only to fail the moment they hit the physical world—where payload shifts, communication latency, joint friction, and actuator wear take their toll.

Diagnosing why a policy broke usually means manually digging through noisy joint telemetry and guessing whether an issue was caused by physics or bad controller gains. We built OmniSim-Agent to automate this entire workflow: an autonomous sim-to-real policy auditor that red-teams robot control policies in headless physics, isolates failure clusters, grounds hardware limits in manufacturer documentation, and outputs a verified safe operating envelope.

What it does

OmniSim-Agent red-teams robot control policies before they are deployed to physical hardware:

  • Adversarial Red-Teaming: Uses NVIDIA Nemotron to generate physics hypotheses (varying payload, speed scaling, control latency, and joint damping) to probe the policy's failure boundaries.
  • Physics Simulation & Continuous Detectors: Runs 240-Hz headless PyBullet simulations evaluating 7 continuous detectors: torque saturation, velocity limits, joint limits, tracking error, Jacobian singularity, collisions, and instability.
  • Bisection Shrinker & Failure Atlas: Groups failures into clusters and bisects parameters toward nominal values to extract the minimal failing reproduction.
  • Tavily Spec Grounding: Searches and extracts official manufacturer documentation (Franka Emika), quote-verifying joint limits against extracted text and flagging URDF model discrepancies.
  • Deterministic Safety Shield & Held-Out Verification: Diagnoses root causes and applies an inverse-dynamics Safety Shield tested across 200 disjoint held-out scenarios—reducing failure rates from 34.5% to 1.5% while preserving a 94% task completion rate.
  • Deployment-Ready Exports: One-click exports of safe_envelope.json, ROS 2 control parameters YAML, an audit HTML report, and an auto-generated pytest regression suite for Robot CI.

How we built it

We structured OmniSim-Agent across an end-to-end modular pipeline:

  • Simulation Core: Headless PyBullet (p.DIRECT) running 240-Hz physics and 60-Hz control with a multiprocess episode runner for deterministic batch execution.
  • LLM Reasoning Layer: Built on Nebius Token Factory using NVIDIA Nemotron models. Nemotron-3.5-Lightning serves as the fast adversary proposing parameter hypotheses, while Nemotron-3-Super powers deep multi-step root-cause diagnosis and mitigation.
  • Knowledge Verification: A Tavily agent that queries official manufacturer domains (docs.franka.de, frankaemika.github.io), extracts page content, and verifies that numeric claims and quotes exist verbatim before admitting them to the envelope.
  • Deterministic Safety Shield: Plain Python/NumPy inverse-dynamics code that caps payload and scales speed dynamically; the LLM never enforces limits directly.
  • Backend & 3D Dashboard: FastAPI with Server-Sent Events (SSE) streaming, SQLite run storage, and a React + TypeScript + React-Three-Fiber dashboard rendering 3D joint torque heatmaps and synchronized telemetry curves.

Challenges we ran into

  • Handling LLM Reasoning Output: Nemotron reasoning models occasionally emit empty content blocks or wrap JSON in <think> tags. We built a defensive parser with balanced-brace extraction and schema-repair retries to ensure reliable structured outputs without depending on native tool calling.
  • Sim-to-Real Limit Discrepancies: Simulator URDF files are often more permissive than real hardware. We built automated verification checks using Tavily to compare URDF limits against manufacturer datasheets.
  • Strict Determinism: Ensuring identical seeds and parameters produce bitwise-identical telemetry across parallel PyBullet worker processes required isolated per-process clients and seeded NumPy generators.
  • Hardware Constraints: Developing on standard PCs without local GPU clusters required relying strictly on hosted APIs via Nebius Token Factory and optimized headless CPU simulation workers.

What we're proud of

  • End-to-End Autonomous Pipeline: Watching the system identify an unhandled torque saturation failure on Joint 6, shrink it to a 2-variable repro, verify official Franka limits, and generate a working ROS 2 parameter file without human intervention.
  • Verified Held-Out Verification: Moving beyond subjective LLM outputs by proving our Safety Shield reduces failure rates from 34.5% to 1.5% on 200 held-out scenarios our adversary never saw during training.
  • Interactive 3D Replay: Building a responsive 3D skeleton visualizer with synchronized torque and velocity telemetry curves that makes complex robotics failures immediately interpretable.

What we learned

  • In Physical AI and robotics, an LLM's opinion is not a safety guarantee; pairing generative hypothesis generation with deterministic physics shields is essential.
  • Grounding AI decisions in manufacturer documentation via Tavily prevents hallucinated hardware tolerances.
  • Structuring team development into strict phased contracts (freezing Pydantic schemas before building UI) keeps complex multi-agent architectures manageable and prevents merge conflicts.

What's next for OmniSim-Agent

  • Reality Anchor: Calibrating simulation damping and latency against uploaded real-world robot telemetry logs using CMA-ES optimization.
  • Multi-Robot Expansion: Expanding beyond the Franka Panda to quadrupeds and mobile manipulators.
  • Robot-CI GitHub Action: Releasing a turn-key GitHub Action that automatically audits control policies on pull requests.

Built With

Share this project:

Updates

Submission history