-
-
Robotic movements
-
stimulation
Inspiration
I work part-time as a shift supervisor at a restaurant, and part of my job is running the stockroom and ordering stock. Moving boxes isn't the hard part. The hard part is the small mistakes: an item on the wrong shelf, a count that's off by two, a delivery put away in the wrong place. Nobody notices until something runs out.
Most robots have the same problem. They follow the plan, make a mistake, and carry on as if nothing happened. I wanted to build a robot that notices when it got something wrong and fixes it without a human stepping in.
What it does
ShelfMind is a simulated stockroom robot. You give it a task in plain English, like "restock shelf B with 6 boxes of item X", and it:
- Plans the task step by step using an NVIDIA Nemotron model
- Acts on the plan inside a Webots stockroom simulation
- Checks what actually happened, using the robot's sensor data and the shelf state from the sim
- Reflects by sending that state back to the model and asking whether it matches the plan
- Corrects itself when something is off, then repeats the loop until the shelf is right
How I built it
- Brain: [Nemotron model name], served through Nebius Token Factory
- World and robot: Webots, with a custom stockroom scene and robot controller
- Editor: Google Antigravity
- The loop: the controller turns the sim state into a short structured summary (robot position, held item, shelf contents) and sends it to the model with the original task. The model replies with either "done" or a corrected next action, which the controller runs.
Challenges
I had never done robot simulation before this hackathon. I started in Unity, spent time fighting it, and switched to Webots partway through, which meant learning a new simulator from zero close to the deadline. The other hard part was getting the model's output into a format the robot could act on reliably. Free-form text plans broke the controller, so I had to constrain the model to a small set of actions.
What I learned
- How robot simulation works in practice: scenes, controllers, sensors, and timing
- That the feedback loop matters more than the first plan. A model that can see its own result and try again does far better than one that only plans once
- How to wire an open model into a real-time control loop through a hosted inference API
What's next
More failure cases (dropped items, blocked aisles, wrong item picked), a proper inventory count check, and eventually testing the same loop on a real robot arm.
Built With
- google-antigravity
- llm
- nebius-token-factory
- nvidia-nemotron
- python
- robotics
- webots
