Retake

Your robot is only as good as its demos; Retake checks every one, flags the bad ones with proof, and tells you what to re-record before you waste a training run!

Comment

Your robot is only as good as its demos

Robots learn tasks by copying human demonstrations. When a demo goes wrong (a spill, a hand in the shot, a sensor that froze), the robot learns that too. Teams record hundreds of hours of demos and then train on all of them, because nobody has time to watch every video.

Retake finds the bad demos for you, shows you the proof, and tells you what to re-record, before you waste a training run.

A human hand in the workspace, caught by Retake This demo ends perfectly, so you'd probably keep it. Retake caught the human hand in the workspace at the start.

Inspiration

While testing my integrity checks on 178 "expert" robot demonstrations, one flag wouldn't go away. I opened the episode: the robot never moves. It was labeled expert.

That made me wonder how many bad demos are hiding in the data people actually train on. So I checked the largest public collection I could find: every episode in LeRobot's community dataset.

What it does

Point Retake at any LeRobot dataset (no video download needed). For a typical dataset, within minutes you get:

  • A retake list: demos to re-record or drop, each with its proof: the failed check, or the frame that shows the problem and when it happened.
  • A review list: demos it isn't sure about. Keeping uncertain calls off the retake list is what makes that list safe to act on.
  • How to fix the cause: likely causes and fixes for each problem found, each backed by a verified quote from LeRobot's own docs or GitHub issues.

Who it helps, and how

  • People recording demos (students, hobbyists, teleoperators): find out which takes failed while the robot and scene are still set up, and re-record only those.
  • Teams training robot policies: stop spending GPU time learning from broken or duplicated demos, and stop duplicates leaking between training and test sets, which inflates evaluation results.
  • Anyone publishing robot datasets: ship data others can trust.
  • Everyone: no more watching hours of video to check data. For LeRobot's community dataset, that's about 365 person-hours of watching, versus about $80 with Retake.

Why it matters

Robot policies learn to mirror their training data. Research on imitation learning shows that policy performance hinges on dataset quality, and that filtering out low-quality demos substantially improves the success rate of the resulting policy (Demo-SCORE, DemInf). Those methods need extra training or robot rollouts. Retake is a check you can run the moment a recording session ends.

What I found in LeRobot's public data

I ran Retake on all of lerobot/community_dataset_v3: 1,754 datasets from 488 contributors, 72,356 episodes, 364.6 hours of robot data, for $0 in model spend.

  • 5,125 episodes (7.1%) are exact motion duplicates of episodes elsewhere in the collection; 777 span different contributors. One pair from two different accounts is identical to six decimal places across 426 frames.
  • 763 episodes (1.05%) are objectively broken: 338 empty, 219 with no motion, 138 duplicated within their own dataset, 71 length mismatches, 56 dead sensors, 15 timestamp errors.
  • In a 52-episode sample, the vision layer flagged 4 demos with a human hand in the workspace. All 4 were confirmed by eye.

Can you trust it?

I froze a labeled test set (50 episodes from BotFails, 25 real failures) before evaluating, tagged the pipeline, and ran it exactly once.

  • Retake list: 13 of 13 correct, zero false alarms (95% lower bound on precision ≈ 77%).
  • Retake + review: 18 of 25 failures surfaced (72%).
  • Integrity checks: 30/30 detection for every injected corruption type, and zero false alarms on 178 clean expert demos.

I diagnosed every miss. Retake is strong on visible accidents (spills, drops, wrong placement, human hands) and weaker on subtle rule-breaks, like a cucumber in the fruit box. Every limit is documented in the README.

Retake test-set report

How I built it

Retake runs in layers, cheapest first, so most problems are caught for free:

  1. Integrity (code, free): 10 deterministic checks, from empty episodes and dead sensors to frozen or black video.
  2. Motion (statistics, free): flags demos that are unusually slow, idle or jerky for their task.
  3. Vision: NVIDIA Nemotron 3.5 Super VL on Nebius Token Factory. Each request shows the model 2 frames from a known-good demo next to 6 frames from the episode and asks for a JSON verdict with evidence. Thinking is off for speed and cost.
  4. Report: Nemotron 3 Ultra, with code guardrails that check every claim in the summary against the data.
  5. Fix guidance: Tavily + Nemotron 3 Ultra, at runtime. Tavily searches LeRobot's docs and GitHub issues; Ultra summarizes using only those sources; code keeps a point only if its verbatim quote is in the cited source, that source describes the same symptom, and it's official LeRobot material.

Every design decision came from a logged experiment (25 in EXPERIMENTS.md, including the failures). Showing the model a known-good reference halved false alarms. Moving uncertain "incomplete" verdicts to the review list took retake precision from 50% to 80% in development. Every model call is cached, budget-guarded and cost-logged: the whole project cost $0.46, and auditing costs about $1.10 per 1,000 episodes.

Challenges I ran into

  • A result that was too good. An early experiment leaked labels into the evaluation. I found it, threw the result out, and re-ran it cleanly.
  • More instructions made the model worse. Twice I gave the vision model task checklists; twice precision dropped (to 56% and 60%, from 80%). The checklist is now off by default.
  • Fix guidance that cited the wrong robot. Early searches returned forum posts about unrelated industrial arms. It took four tightening passes before the guidance could be trusted. Where nothing survives, Retake says so instead of guessing.

Accomplishments that I'm proud of

  • A frozen test set, run once, with zero false alarms on the retake list.
  • Auditing an entire public robot-data collection for $0, and finding hundreds of problems no one had flagged.
  • An experiment log that documents five negative results as carefully as the wins.

What I learned

  • Vision models judge best by comparison: one good reference demo beats a paragraph of instructions.
  • The useful question isn't "can the model detect failures?" but "which of its calls can you act on without checking?" That's why Retake has two lists.
  • Retake's misses share one pattern: the model checks that the end state looks right, not every rule of the task. That's what to fix next.

What's next for Retake

  • Prove the payoff directly: train the same policy on raw and Retake-cleaned data, and measure the difference in success rate.
  • Catch subtle rule-breaks: focused per-rule checks, smarter camera choice, denser frames.
  • Run where data is made: automatically after every recording session, so bad takes are caught before anyone leaves the lab, and inside teams' own clouds for private data.

Built With

  • hugging-face
  • lerobot
  • nebius
  • nebius-token-factory
  • nemotron-3-ultra
  • nemotron-3.5-super-vl
  • numpy
  • nvidia-nemotron
  • openai-sdk
  • pandas
  • pillow
  • pyarrow
  • python
  • scikit-learn
  • tavily
Share this project:

Updates

Submission history