RoboLab OS - Autonomous Robotics Research Engineer

autonomous research engineer agent that helps you experiment and iterate through your robotics projects development

Comment

Inspiration

One of the biggest bottlenecks of developing robotics application these days is how long it took for the devs or engineers to go from simulation to real world application because of the massive difference in the environments, it could be lightning or friction or anything slightly different will most probably make the robots fail in real world, so the engineers manually tweak parameters, re-run tests, and repeat for weeks.

To battle these bottlenecks, I built an autonomous robotics research agent that works as engineers' assistant so they don't have to manually tweak the parameters and 'just' wait for the results. The autonomous agent can manages the entire lifecycle of robotics task using Interactions API available via Gemini 3 to run long-term experiments and Action Mode to execute the simulation scripts needed to achieve its target state.

It's Claude Code for robotics experiment and development.

What it does

Target Users:

  • Robotics Engineer/Researcher

Features:

  • Basic Simulation
  Run a 10-second hexapod simulation on sand terrain

  Simulate a drone flight for 30 seconds with 5 m/s wind speed in gusty conditions
  • Video Analysis
  Analyze the video from the last run, focus on the landing and suggest on how can we do the landing better and safer for the robot.
  • Autonomous Research (multi-iteration)
  Fix the hexapod gait instability on sand — run autonomous research with success criteria stability > 95%

  Optimize the drone hover stability in gusty wind conditions. Run up to 8 iterations."

  Optimize the drone’s landing PID for 15mph wind conditions.

How we built it

  • Backend & Database: Express.js & Cloudflare R2
  • Frontend: React + Vite (TypeScript)
  • AI Functionality: Gemini 3
    • Interactions API

Challenges we ran into

  • Datasets: trying to combine the unstructured datasets such as csv files for telemetry and images for visual camera feed has been challenging

  • Interactions API: Working with the Interactions API was a bit confusing as it was still quite new.

  • Making sure that Gemini 3 is doing the long-running tasks correctly whilst also maintaining its context across sessions and iterations of experiments.

Accomplishments that we're proud of

1. Gemini 3 Interactions API

Built an autonomous research mechanism that runs a long-running experiment in the background and able to do multi-step tool calls to achieve its target.

2. Multimodal capabilities

Utilizes Gemini 3 ability to ingest images/videos and texts (csv files) to then analyze several unstructured data sources in sync.

3. Parallel agents run

With the help of Gemini 3 Interactions API, the user can spun several runs of the agent one go to manage several robots connected to the system.

What we learned

  • Gained better understanding of how powerful Gemini 3 is with long-running background tasks - extending the capabilities of usage of AI in the modern era of agents.
  • Interactions API is a real game changer with the new era of Action Era as it enables devs to develop a products that just do things also the ability to run them in background too.

What's next for RoboLab OS - Autonomous Robotics Research Engineer

  • Sim-to-real mechanism where RoboLab OS can truly build a simulation based on the real-world runs that the robot has done with the Genie 3 world model rather than using simple simulation mechanism such as ROS.

  • Real-time configuration tuning: Ability to tune the physics or motor configuration of the robots whilst doing real-time runs/experiments on the robot.

  • Fleet management: Ability to manage a fleet of robots such the one you can find in warehouses and display their data as a group.

  • Human-in-the-loop (HITL): Let the users interfere with the experiment midway if they see something is wrong or they want to tweak small parameters

  • Run real simulation: As currently the simulation is still very simple - I want to seriously pursue this path for the product by enabling a server-side cloud simulation where the user will be able to see the visualization of the simulation too.

Built With

Share this project:

Updates