-
-
cover
-
results
-
architecture
-
reliability
Inspiration
Training robot imitation-learning policies produces many checkpoints, but deciding which checkpoint to deploy is surprisingly difficult. A low validation loss does not guarantee successful robot behavior, because the robot's own actions influence its future observations. Evaluating every checkpoint through physical robot rollouts is expensive, and real robots often lack a programmable success detector.
My earlier co-authored research, "Cheap Metrics First, Replanning Diagnostics Second: Offline Checkpoint Screening for Robot Imitation Learning," studied 1,365 checkpoints across 45 training runs. That study showed that inexpensive offline metrics could help screen candidate policies, but more accurate learned dynamics did not necessarily lead to better checkpoint selection.
For the Nebius x NVIDIA Global AI Hackathon, I wanted to go beyond offline metrics and answer two practical questions:
Can NVIDIA's open models evaluate what physically happened during a robot rollout, reducing the need for manually programmed success detectors?
And more importantly, can I trust those AI judgments when making decisions about robot policies?
These questions led me to develop Cosmos-PLV as my hackathon project.
What it does
Cosmos-PLV is a working robot policy evaluation and AI-judge auditing system.
It combines inexpensive checkpoint screening, real robot simulations, NVIDIA visual models, and Nebius Token Factory inference in one interactive Streamlit application.
The workflow consists of five stages.
Stage 1: Cheap checkpoint screening
The system evaluates Diffusion Policy checkpoints using PLV metrics from my earlier research, including action prediction error, temporal smoothness, model-feedback smoothness, and replanning consistency.
These metrics can be calculated without running expensive evaluation rollouts.
Stage 2: Robot simulation
Promising checkpoints are evaluated in robosuite and MuJoCo. The system records the robot's actions, predicted action chunks, observations, and execution videos.
Each simulation also provides ground-truth success information for independent verification.
Stage 3: NVIDIA visual evaluation
I use two NVIDIA open models.
NVIDIA Cosmos3-Edge runs locally on an RTX 3090. It examines individual frames and describes physical object states, such as whether a can is being grasped, lifted, or placed.
NVIDIA Nemotron-3.5-Super-VL runs through Nebius Token Factory. It receives robot execution videos and produces structured success and failure judgments.
The application records model identifiers, request IDs, inference latency, and token usage.
Stage 4: Checkpoint recommendation
The dashboard combines inexpensive screening results with visual judgments to help researchers determine which checkpoints deserve further evaluation.
Importantly, the system also displays a free baseline: simply selecting the latest training epoch.
I report cases where the simple baseline performs just as well as the more complicated methods in my system.
Stage 5: AI-judge reliability auditing
The system compares AI-generated judgments against verified simulator outcomes.
It highlights disagreements, identifies systematic failure modes, and warns users when seemingly accurate AI judgments may be unreliable.
Users can inspect real rollout videos, trigger Cosmos and Nemotron inference, review evaluation results, and explore the failure cases that motivated this project.
How I built it
I built Cosmos-PLV using Python, PyTorch, Diffusion Policy, robomimic, robosuite, MuJoCo, NVIDIA Cosmos3-Edge, NVIDIA Nemotron-3.5-Super-VL, Nebius Token Factory, and Streamlit.
I organized the implementation into two main software environments.
The first environment handles robot policy training, checkpoint screening, and simulation. The second handles NVIDIA visual inference, Nebius API requests, and the interactive dashboard.
The environments exchange recorded videos, state data, and structured JSON results.
I retrained 93 Diffusion Policy checkpoints across three training runs on the Can and Square manipulation tasks. Each checkpoint was evaluated using 50 training-time simulation rollouts, producing 4,650 rollout results. I additionally recorded 330 evaluation episodes with video for my main NVIDIA model comparison.
Cosmos3-Edge runs locally in BF16 precision on a single RTX 3090 with 24 GB of VRAM.
Nemotron-3.5-Super-VL is accessed through Nebius Token Factory using an OpenAI-compatible API. I send robot videos to the model and validate its structured responses.
Across my experiments, I logged 7,005 Nebius API calls with zero recorded API failures, at an estimated inference cost of approximately $4.96 using published pricing. This is a pricing estimate, not an invoice.
Nebius allowed me to use a powerful video-language model without deploying or maintaining a second GPU server. My local GPU remained available for robot simulation, training, and Cosmos inference.
I also developed a Streamlit dashboard with live model calls and a recorded replay mode. The replay mode lets reviewers inspect real recorded executions and model responses without an NVIDIA GPU or Nebius API key.
What I measured
1. NVIDIA models can judge robot success from video
I evaluated the models on 330 recorded robot simulation episodes, using privileged simulator success labels as ground truth.
Nemotron-3.5-Super-VL achieved 96.4% success-classification accuracy, with a 95% confidence interval of 93.8% to 97.9%.
Its balanced accuracy was 92.7%.
Cosmos3-Edge achieved 91.8% accuracy, with a 95% confidence interval of 88.4% to 94.3%.
Its balanced accuracy was 83.2%.
However, these results do not mean that the models are reliable in every situation.
Nemotron incorrectly classified 9 of 67 failed episodes as successful in this evaluation set. Cosmos incorrectly classified 21 of those 67 failures as successful.
Furthermore, detailed failure-type classification remained unreliable for both models.
2. Evaluation can be made more efficient
I explored several approaches to reducing the cost of evaluating robot policies.
In exploratory, post-hoc analyses, adaptive rollout allocation required approximately 2–3 times fewer rollouts than uniform allocation to reach a comparable checkpoint-selection quality under the tested evaluation budget.
This improvement came from adaptive allocation and a strong latest-epoch prior, not from a unique advantage of PLV.
A time-limit early-abort strategy reduced summed simulation execution time by approximately 16% on 200 evaluation episodes, while prematurely stopping 2 of 141 successful attempts.
Using privileged simulator information increased the estimated saving to approximately 20%, although that information would not ordinarily be available on a physical robot.
A local-Cosmos-first evaluation cascade also reduced Nebius API calls by approximately 16%, with a small decrease in classification accuracy.
These are exploratory efficiency results, not claims of faster policy training or universally superior checkpoint selection.
3. Simple checkpoint-selection baselines are surprisingly strong
In my reproduced experiments, PLV metrics correlated well with robot success.
The best PLV-family metric achieved a mean within-run Spearman correlation of approximately 0.801, and the metrics retained near-optimal checkpoints within their top 30%.
However, simply choosing later training epochs achieved a correlation of 0.815.
No PLV metric outperformed this free baseline in my newly trained checkpoint pool.
My training runs were relatively short and nearly monotonic, meaning that later checkpoints generally performed better.
I explicitly report this limitation instead of presenting the screening method as an improvement over a baseline it did not beat.
What surprised me: an accurate AI judge can still be dangerous
The most important finding of my project was not the overall accuracy of the NVIDIA models.
It was discovering what happened when I used those judgments to select additional robot training data.
Experiment c1: A systematic visual blind spot
I conducted a pre-registered reliability audit involving 620 additional Can-task rollouts from 31 checkpoints.
The robot's objective was to place a red can in the correct compartment of a destination bin.
In some failed episodes, the robot moved the can into the destination bin but placed it in the wrong compartment.
To a human observer, this is an incorrect placement. The simulator also records it as a failure.
Yet Nemotron frequently judged these episodes as successful.
Nemotron classified 32 of 33 wrong-compartment failures as successes.
Cosmos3-Edge also exhibited this failure mode.
Even after I explicitly described the correct compartment in Nemotron's prompt, it still accepted 24 of those 33 failures.
This revealed a systematic weakness: recognizing that the can reached the general goal area was easier than verifying that it reached the exact required location.
As policies improved, obvious missed-grasp failures became less common, while near-goal placement failures made up a larger fraction of the remaining failures.
Consequently, overall success-classification accuracy could appear strong while important mistakes remained undetected.
I also found that the strength of this trend depended on a small number of initial states, limiting how broadly it can be generalized.
Experiment s2: What happens when I train on AI-selected successes?
I then investigated whether Nemotron's success judgments could improve a weaker Diffusion Policy.
The idea was straightforward: run the policy, identify successful attempts using Nemotron, and add those attempts to the training dataset.
I compared this approach against continued training, simulator-verified success selection, and additional human demonstrations.
All evaluated policies were tested on the same 500 initial evaluation seeds.
Continued training achieved a success rate of 85.6%.
Adding 36 human demonstrations increased success to 93.8%, an improvement of 8.2 percentage points.
Adding the policy's own simulator-verified successful episodes produced 82.4% success, which did not demonstrate an improvement over continued training.
Adding episodes that Nemotron classified as successful produced only 75.2% success.
This was 10.4 percentage points below continued training, with a paired 95% confidence interval for the difference of -14.8 to -6.0 percentage points.
Further investigation showed that Nemotron had accepted 58 of 149 genuinely failed episodes as successful. Of those 58 false successes, 44 involved the same wrong-compartment failure identified in my earlier audit.
These mislabeled episodes entered the self-training dataset.
The experiment demonstrates a serious practical concern: high overall classification accuracy does not guarantee that AI-generated labels are suitable for training robot policies.
I do not claim that every part of the observed performance decrease was caused by Nemotron's labeling errors. The experiment used one fine-tuning run per condition, and differences in training-data composition and dilution of the original demonstrations could also contribute.
Nevertheless, the results clearly show why AI-generated training labels require independent reliability checks.
Experiment r1: Can AI recognize beneficial replanning?
I also tested whether the models could determine when a robot should replace an earlier action plan with a newly generated one.
At 400 replanning boundaries, I restored identical simulator states and evaluated the consequences of executing the old and new plans.
The models did not reliably predict which plan would lead to a better outcome.
PLV's replanning disagreement score identified fragile episodes, but it did not reliably identify the exact replanning decision responsible for changing the outcome.
This negative result reinforces an important distinction: detecting uncertainty or inconsistency is not the same as identifying a better action.
Challenges I faced
Grounding visual reasoning in observed evidence
An early Cosmos prompt caused the model to narrate the robot's intended actions instead of describing what actually appeared in the video.
By switching to task-free questions about observable object states, I obtained more useful physical observations.
Understanding model limitations
My initial action-conditioned world-model experiment used an action slot that was not trained in the released Cosmos3-Edge checkpoint.
After identifying the problem, I retracted that result and tested the supported Franka action representation.
Although the generated motion showed some directional agreement with recorded robot movements, it was not sufficiently accurate to replace actual simulator rollouts.
Managing compute resources
I performed robot training, simulation, and local Cosmos inference on one RTX 3090. Separate environments and careful GPU scheduling were required.
Ensuring scientific reliability
I used fixed evaluation protocols, separate prompt-development data, paired evaluation seeds, independent result verification, and pre-registered follow-up experiments.
I also documented negative findings, post-hoc analyses, protocol amendments, and limitations rather than reporting only favorable results.
What I learned
My main lesson is that AI evaluation quality must be measured by the consequences of its errors, not just by aggregate prediction accuracy.
Nemotron performed well as a general video-based success detector, but its systematic failures could contaminate downstream training data.
I also learned that sophisticated checkpoint metrics do not automatically outperform simple baselines, and that a visually plausible world-model prediction does not necessarily provide useful feedback for choosing robot actions.
The practical outcome is a more cautious approach to robot evaluation:
I use inexpensive methods where they are sufficient, NVIDIA visual models when they provide additional information, and explicit audits of systematic mistakes before relying on AI-generated judgments for important decisions.
What is new during the hackathon
This project builds upon earlier PLV research, which existed before the hackathon submission period.
I do not claim the original PLV research or its historical 1,365-checkpoint results as new hackathon work.
During the hackathon, I developed the Cosmos-PLV application, including:
- A new simulation recording and evaluation pipeline.
- NVIDIA Cosmos3-Edge integration on a local GPU.
- Live Nemotron-3.5-Super-VL integration through Nebius Token Factory.
- Structured diagnostic outputs and API usage logging.
- A dashboard for checkpoint selection, video review, and AI-judge auditing.
- A newly trained pool of 93 Diffusion Policy checkpoints.
- New model comparison and evaluation efficiency experiments.
- Pre-registered replanning, judge-reliability, and self-training audits.
- A recorded replay mode for reviewers.
I do not claim to have invented VLM success detection, action-chunk consistency monitoring, active policy evaluation, or robot self-improvement. These areas have established prior research, including SuccessVQA, AutoEval, Sentinel, SOAR, and RoboCat.
My contribution is the integrated, reproducible evaluation workflow and the experimental analysis of when its visual judgments are useful and when they become unreliable.
What's next
I plan to extend Cosmos-PLV to more robot tasks, camera viewpoints, and eventually real robotic systems where privileged simulator success labels are unavailable.
I also want to investigate spatially grounded visual verification that checks exact placement locations rather than relying on general scene-level judgments.
Another direction is to improve action-conditioned Cosmos predictions through robot-specific post-training, potentially reducing the need for some physical rollouts in the future.
These are future directions, not completed features.
My long-term goal is to make robot policy evaluation more efficient, transparent, and trustworthy.
Cosmos-PLV demonstrates that evaluating robots with AI requires more than an accurate judge. It requires understanding exactly when that judge is wrong.
Demo and source code
YouTube demo: https://youtu.be/XS5KSOcLfC8
Public GitHub repository: https://github.com/cheesss/cosmos-plv
The repository includes source code, setup instructions, experimental protocols, result files, and an MIT license.
A recorded replay mode is available for reviewers who do not have access to an NVIDIA GPU or Nebius API key.
