-
-
Our real robot: a mobile base with an SO-100 arm and basket, running NVIDIA GR00T N1.7 on a Jetson AGX Orin.
-
Nemotron Nano reads the order, Super plans each step, safety checks guard it, GR00T picks and Ultra writes the report.
-
Plain-language orders: Nemotron Nano turns "2 biscuits, one fevical and a screen cleaning gel" into products, even with a typo.
-
Live mission: Nemotron Super decides every drive and pick while the robot moves; each decision and its result is logged.
-
One finished mission: three Nemotron models and GR00T, step by step, ending with a mission report by Nemotron Ultra.
-
The real shelf: upper level holds ink box, Fevicol and screen gel; lower level holds Frooti and biscuits, as on the map.
StorePilot AI: A dark-store picking robot planned by NVIDIA Nemotron and picking with GR00T.
About the Project
Imagine placing an order and having a robot go around the store, find the required products, pick them up, and bring them to the packing area. That's what we're trying to build with Store-Picker-Bot.
We started this project with a simple idea: instead of making robots follow a fixed set of commands, what if we could tell them what we need in plain English and let AI help plan the work?
For example, a store operator could enter, "Pick two packets of Maggi and one Coke bottle." The system would turn that request into a set of tasks for the robot to execute.
How We're Building It
We're combining AI with a real mobile robot to make this possible.
We're using NVIDIA Nemotron through Nebius Token Factory to understand orders and generate structured task plans. The Jetson Orin handles the robotics side, including ROS 2, Nav2, and the integration of sensors and navigation. We're also working with a Raspberry Pi 5B and a touchscreen LCD to build a dashboard where the operator can submit orders and monitor the robot.
For product picking, we're working towards integrating a camera with the SO100 robotic arm using LeRobot. The aim is to let the robot navigate to a shelf, identify the required item, pick it up, and place it in its basket.
We want the dashboard to show what's happening throughout the mission, from the initial order to the final result, rather than making the operator guess what the robot is doing.
Why We Chose This Approach
One of the interesting parts of this project is figuring out how to connect an AI model to a physical robot reliably.
An AI model can understand an instruction and suggest a plan, but the robot still needs proper navigation, perception, motion control, and safety checks to carry it out. We're keeping these responsibilities separate so that the language model doesn't directly control the motors or robotic arm.
We're using Nebius Token Factory for hosted model inference instead of trying to run a large language model on the Raspberry Pi. This lets us keep the dashboard lightweight while the Jetson focuses on robotics workloads.
Challenges We're Working On
Getting the different parts to work together is one of our biggest challenges. The AI needs to produce tasks that the robot can understand, the navigation system needs to reach the correct location, and the manipulation system needs to handle products reliably.
We're also working on connecting the Jetson's robot status to the Raspberry Pi dashboard so that the information displayed on the screen reflects what is happening on the actual robot.
Our approach is to build and test each part separately before bringing everything together.
What's Next?
We want Store-Picker-Bot to become more than a robot that moves from one point to another. Our goal is to demonstrate a complete workflow where a store operator gives an order, AI helps plan it, and a physical robot carries out the task.
We're building towards a working demonstration of autonomous navigation and product picking, with a clear interface that makes the entire process easy to understand.
Built With
- fastapi
- lerobot
- nav2
- nebiustokenfactory
- nemotron
- nvidia
- nvidiajetson
- python
- raspberry-pi
- react
- ros2
