
Inspiration
Robots can perform complex physical tasks, but translating human intent into reliable robot actions remains a significant challenge. We wanted to explore how AI could bridge this gap by allowing users to describe a task in natural language rather than manually specifying every movement. This inspired RoboPilot-Sim2Real, a simulation-first robotics framework that connects large language model reasoning with robot-specific control and physics-based simulation. Our initial focus is robot manipulation: enabling a simulated robot to understand a pick-and-place task, generate a structured plan, and attempt to execute it in a controlled environment.
What it does
RoboPilot-Sim2Real transforms natural-language manipulation instructions into structured plans that a simulated robot can attempt to execute.
For a pick-and-place task, the system follows this workflow:
Understand the task: Receive a natural-language instruction and the current scene observation.
Generate a plan: Use NVIDIA Nemotron using Nebius Token Factory to produce a structured JSON plan with high-level actions.
Validate the plan: Check the task, supported objects, destination, action sequence, and recovery policy.
Execute the actions: Pass the validated plan to a bounded, robot-specific controller operating in MuJoCo.
Evaluate the outcome: Use simulator observations to determine whether the intended manipulation succeeded.
The current task sequence includes approaching an object, grasping it, lifting it, moving it to a destination, releasing it, and verifying placement. Rather than allowing the language model to generate unrestricted motor commands or joint angles, the system separates high-level planning from robot control. This creates a foundation for safer, more modular experimentation with AI-driven manipulation.
How we built it
We built RoboPilot-Sim2Real in Python, integrating NVIDIA Nemotron through an OpenAI-compatible inference API and MuJoCo for physics-based robot simulation.
The architecture consists of several key components:
AI planner: Converts instructions and scene observations into structured action plans.
Plan validation layer: Parses model responses, normalizes recognized action aliases, and enforces the expected schema and action sequence.
Robot profile system: Stores robot-specific configuration, including model assets, joints, actuators, home positions, and gripper settings.
Simulation environment: Loads the robot’s MJCF model and generates the task scene, including the table, manipulation object, and destination.
Robot-specific controller: Attempts to execute the validated actions using bounded control logic.
Observation and verification: Captures robot and object states to support execution monitoring and outcome assessment.
We used a modular, incremental development approach. We first addressed robot-model loading and mesh-path compatibility, then integrated the inference API, resolved structured-output mismatches, and progressed to executing validated plans in simulation.
We leveraged a combination of AI models, AI-powered research tools, and simulation technologies while developing RoboPilot-Sim2Real.
- NVIDIA Nemotron (NVIDIA-Nemotron-3-Nano-30B-A3B model via Nebius Token Factory)
- Used as the primary high-level reasoning and task-planning model.
- Interprets natural-language robotic manipulation instructions and translates them into structured action plans.
- Uses prompt engineering and constrained planning instructions to generate outputs compatible with our predefined robot control primitives.
- The generated plans pass through deterministic validation before execution in MuJoCo.
- Tavily (Search API)
- Used as an AI-oriented web search and information retrieval tool.
- Helps retrieve relevant and robot-specific external technical information, documentation, and research references during model inference.
- Operates as context builder with research and information gathering around robotics, simulation, AI models, and related technical concepts.
- Complements Nemotron as a tool providing access to external information.
- MuJoCo (Physics Simulator)
- Used as the physics-based robotics simulation environment.
- Supports robotic manipulation, environment interaction, simulation-based testing, and execution verification.
- ChatGPT (AI Assistant)
- Used as an AI-assisted development and research tool.
- Supported architecture design, implementation guidance, debugging, code refinement, and technical documentation.
- Assisted with exploring robotics, reinforcement learning, and simulation-driven development approaches.
Our architecture deliberately separates language-based reasoning from physical execution and policy training. Nemotron handles high-level task planning, Tavily supports external information retrieval, and deterministic controllers with MuJoCo govern simulated robot execution. As extension to the hackthon project, we have also added the training pipeline that remains independent of LLM-generated instructions or training data.
Challenges we ran into
Robot asset integration: MuJoCo initially failed to locate a referenced Franka Panda mesh because the expected path did not match the staged asset directory. We investigated the MJCF configuration and temporary asset layout to address the loading issue. Inference timeouts: Our standalone Nemotron request worked, but the application initially timed out. Comparing the two implementations revealed differences in endpoint configuration and timeout/retry behavior. We aligned the application with the working setup and added timing logs for diagnosis. Inconsistent AI output: The model occasionally returned abbreviated action names such as approach instead of approach_object. We strengthened the system prompt and introduced controlled alias normalization while retaining strict plan validation. Trajectory convergence: After the planner began producing valid plans, execution exposed a deeper robotics challenge. The controller attempted to move the Franka Panda toward the object but failed its joint-trajectory convergence check. This demonstrated that generating a valid action sequence is not sufficient: the controller must also produce feasible motion and successfully execute the grasp. These challenges helped us separate inference, planning, validation, simulation, and control failures instead of treating the entire pipeline as one opaque system.
Accomplishments that we're proud of
Integrated NVIDIA Nemotron into a robotics workflow that accepts natural-language task instructions.
Established a structured planning interface with explicit task, object, destination, action, and recovery fields.
Implemented validation between AI-generated plans and robot execution.
Integrated the Franka Panda model into a generated MuJoCo task environment.
Diagnosed and resolved multiple development-stage issues involving mesh paths, API configuration, and action-schema mismatches.
Progressed from initial simulation setup to executing validated plans through the robot controller.
Established a modular foundation for extending the framework to additional robot profiles and manipulation scenarios.
We are particularly proud of the separation between AI reasoning and robot-specific execution. It allows each layer to be inspected and improved independently, which is essential when developing systems that interact with the physical world. The full pick-and-place task is still being refined; reliable trajectory convergence and successful placement verification remain important milestones.
What we learned
We learned that AI-driven robotics requires more than a capable language model. The system must connect reasoning with physical constraints, structured interfaces, reliable control, and environment-based verification.
Our key takeaways are:
A plan is not a trajectory. High-level actions must be translated into feasible robot motion.
Structured output needs validation. Prompts improve consistency, but the application must enforce its own requirements.
Simulation makes failures observable. Robot states, object positions, and execution histories help distinguish planning errors from control failures.
Modularity accelerates debugging. Separating the planner, validator, robot profile, simulator, and controller makes individual failures easier to isolate.
Success must be verified. A model cannot establish task completion simply by generating a plausible response; the simulator must confirm the outcome. These lessons have shaped our approach toward a more inspectable and extensible robotics framework.
What's next for RoboPilot-Sim2Real
Our immediate priority is to improve trajectory convergence for the Franka Panda and complete a reliably verified pick-and-place cycle. This includes investigating inverse kinematics, joint-control behavior, target reachability, grasp execution, and placement verification.
We then plan to:
Improve motion planning and recovery behavior.
Test manipulation under different object positions and scene configurations.
Expand robot-profile support to additional simulated robots.
Strengthen task evaluation using measurable simulation outcomes.
Explore synthetic-data generation and learning-based control as future extensions.
Investigate sim-to-real transfer once the simulation pipeline is sufficiently reliable.
Our long-term vision is to make RoboPilot-Sim2Real a flexible experimentation platform for connecting language-model reasoning, robot control, and simulation-based learning—helping developers explore how natural-language intent can become dependable physical action.

