Inspiration
I want robots to work toward a goal when their surroundings change, without programming every possible situation in advance. Before building that larger vision, I wanted a small example that I could inspect: what did the robot see, what did it do, what remains uncertain, and did the task actually happen?
SiteShift explores those questions in a simulated kitchen. The longer-term goal is more adaptive planning. The current prototype studies a limited part of it: inspection, uncertainty, a bounded next-step choice and verification.
What it does
A simulated Franka Panda arm must find a sponge, pick it up and scrub a bowl. Its site file says the sponge is in drawer 0. You can run three situations: nothing moved; the sponge and a spray bottle swapped drawers; or a towel covers the sponge.
The robot inspects the kitchen using simulated depth and object labels. Each area becomes sponge seen, sponge not found in the checked zone, or still to check. In the moved case it picks from the drawer where it perceived the sponge. In the hidden case it asks for help rather than declaring the sponge absent.
The visual launcher starts a new MuJoCo calculation on your Mac. You see an overview and the robot’s two cameras while it runs. Replay movements plays the saved images more smoothly after the calculation. Run analysis reads that new run’s evaluator records and compares the episode with its recorded reference.
A separate Recorded evidence page shows the historical episodes and checks their reports claim by claim. That report is written by code from robot-side records, then checked against simulator records the robot never receives. In those three reports, 22 of 39 claims are confirmed, none are contradicted and 17 cannot be checked. These are counts from development examples, not a reliability rate.
How we built it
The simulation uses MuJoCo, NumPy and MuJoCo Menagerie’s Panda model. The kitchen, inspection program, task gestures, guards and evaluator are written in code. The robot does not yet produce an open-ended plan, and it has not been tested on physical hardware.
NVIDIA Nemotron 3 Super, through Nebius Token Factory, chose between scrub_bowl and ask_help when both options were valid. Its input was the robot’s state as text, with no images. Code checked its answer before executing the selected program. Two real calls were recorded for the main demo; when only ask_help was valid, a rule chose without a model call.
The public visual demo recalculates physics and reuses a recorded answer only when the request text matches exactly. It makes no live model call. The repository includes a separate client for reconstructing the request, with a dry run by default and an optional live call using the tester’s own credentials and budget.
A second pilot ran an existing AssembleNIST peg insertion policy unchanged in a CPU-only Nebius AI Cloud Serverless Job. It produced 960 episodes across three conditions. Its purpose was to compare success observed during an episode with the state at report time and examine the limits of a robot-side report. We found 32 disagreements about whether a part was held and disclose them. No Nemotron, retraining or adaptation was used in that pilot.
Who this could help
Robotics developers and integrators testing manipulation tasks. A possible use is to reproduce a change, inspect a failure and compare revisions. This is an intended audience, not validated customer demand. Compatibility with arbitrary models and robots has not been demonstrated.
Challenges and what we learned
A missing observation is not evidence of absence. Keeping an area still to check matters as much as finding the object. A completed motion or a model’s answer also does not establish that the physical task succeeded; the evaluator needs separate records.
The visual launcher was added so another person can run the episode and see the arm move, instead of only reading logs. Its spectator renderer leaves the frozen robot source unchanged, and each new episode is compared with the reference facts. The three situations matched on the development Mac. That remains a reproduction on one machine.
Limits and next steps
Everything here is uncalibrated simulation. The main task uses one development kitchen, a Panda arm, simulated perception and programmed gestures. Nemotron makes a small choice between two programs; the examples do not measure its advantage over a fixed rule.
The robot cannot lift cloth. A separate diagnostic found an unresolved false-absence issue with a thin cloth reported as the catalog towel. The calibration contact is exempt from the bench’s contact rule despite exceeding 20 N. The second pilot uses simulator pose rather than camera perception, and its raw signals were not exported. Its audit can be recounted from the exported data, not recomputed from those raw signals.
Next steps would be broader tasks and environments, a comparative evaluation of the model choice, and perception from camera images. Those are future work.
Try it
The visual simulator currently requires macOS and was tested on one Apple silicon Mac. Clone the public repository, run sh setup.sh, then ./start.command, and open http://127.0.0.1:8766/. Select a situation and click Play simulation. No API key, GPU or cloud service is needed for this local reproduction. Other platforms can view the recorded evidence website described in the README.
