Showhand

Teaching a humanoid a move? Record one take and Showhand names the seconds to show again. On my three takes it sent back 4 of 20 seconds, measured on a simulated Unitree G1.

Comment

Who it's for: someone recording a move to teach a humanoid robot, right before that take goes into a training set. Showhand watches a simulated Unitree G1 copy the take and names the seconds to show again. On my three takes it sent back 4 of 20 seconds, and every one of them points at the measurement or the vision check that failed.

Watch it. https://showhand-replay.vercel.app plays each of my three phone takes beside the simulated G1 that replayed it, on one clock, with every second's grade from physics and from two vision models, each Nemotron answer the guard refused or passed, and the seconds sent back. No login, no key.

Check it. No robot, GPU or key is needed to rerun the grading on my records:

git clone https://github.com/StephenSook/showhand && cd showhand
python -m venv .venv && .venv/bin/pip install -e ".[dev]" && .venv/bin/pytest -q

That's 88 tests, offline.

Inspiration

From August 21 to September 27, Ultimate Bots ran GHOST TRIAL 04, the SuperSONIC Challenge, powered by Nebius: teach a Unitree G1 a move NVIDIA's stock SONIC controller can't do yet. Entrants build a motion dataset, fine-tune the released SONIC checkpoint, and show the move landing in simulation. Each one gets $50 of Nebius compute, and the FAQ's advice is short: "Spend your effort on data." (https://www.ultimatebots.com/hackathon)

That's the part I wanted to help with. A take the robot can't follow wastes the fine-tune it goes into, and you usually find out after the compute is spent. So before a take goes into a dataset, the person who recorded it should know which seconds the robot can't track, why, and what to show again. That's the whole job of Showhand. It doesn't turn video into robot commands. NVIDIA's stock stack does that part, and I built everything that grades what comes out.

What it does

Record. You film one in-place move on a phone.

Replay. NVIDIA's stock path drives a simulated G1 through it: GEM-X pulls the body motion out of the video, the SOMA retargeter maps it onto the G1, and GEAR-SONIC controls the G1 in MuJoCo physics.

Measure. Showhand cuts the replay into one-second windows and checks each against thresholds I froze in git before the first graded run: joint tracking error, root height, tilt, foot slip, time out of balance and falls.

Look. Two vision models compare the human and the robot for each second: NVIDIA Cosmos-Reason1-7B in a Nebius AI Cloud job, and Nemotron 3.5 Super VL on Nebius Token Factory. Both get the same fixed question: does the robot's pose match?

Decide. nvidia/Nemotron-3_5-Lightning on Token Factory reads the measurements and the Cosmos verdicts and answers accept or re-show, with the exact seconds and the failure code behind each one. A guard written in plain code refuses any answer that accepts a take against failing evidence, sends back a second with no failing evidence, or gives a reason that isn't a cited failure code. If Nemotron is refused twice, Showhand re-shows every failing second itself.

Show. The replay page leads with the answer: accept or re-show, and which seconds to film again, also marked on the video's seek bar. The phone video and the robot run on one clock. Click any second and you get its measurements against the limits, both vision verdicts in the models' own words, and how the decision was made. A panel beside the video says where each stage of that take ran.

My three takes

I recorded three takes of one in-place move (184, 187 and 178 frames at 30 fps) and ran each one through the whole pipeline.

  • The robot stayed up. Zero falls on all three. SONIC got every frame: 306, 311 and 296 published at 50 Hz, zero missed deadlines, worst jitter 0.10 ms. Mean joint tracking error was 0.24, 0.22 and 0.23 rad.
  • All three still fail the whole-take cutoffs, for specific reasons. take_0525: foot slip and 1.28 s out of balance. take_0526: 0.55 s out of balance against a 0.50 s cutoff. take_0527: 0.53 s out of balance, and not one of its seconds fails on its own. It only crosses the line when the seconds add up.
  • Cosmos matched 6 of 7, 7 of 7 and 5 of 6 seconds. Its two mismatches, in take_0525 and take_0527, both list orientation.

Then fusion. Nemotron's first answer was refused on every take:

  • take_0525, deterministic fallback. Both Nemotron answers proposed re-showing 2 to 3 s, a second with no failing evidence, and the guard refused both. Showhand re-showed every failing second instead: 1 to 2 s (worst joint error 1.411 rad against a 1 rad limit) and 4 to 5 s (a pose mismatch from Cosmos).
  • take_0526, passed after one retry. The first answer accepted the take against failing evidence. Told why it was refused, the retry passed: re-show 0 to 1 s (out of balance).
  • take_0527, passed after one retry. Same first refusal. The retry re-showed 3 to 4 s for a Cosmos pose mismatch, in a second the physics had passed.

Fusion for all three takes cost $0.00087 on Token Factory, and every request id is in the records.

Then a human graded them. I emailed the three clips to an outside robotics builder and asked for keep or redo. They kept all three. So on its first outside test, Showhand disagreed with a human 3 times out of 3: the grader let every take through, and so did tracking error alone, while Showhand sent seconds back on each one for balance, foot slip, a joint-error spike or a Cosmos pose mismatch. My pre-registration says that when a judge's kappa is undefined (here the human and tracking error gave every take the same grade), the result is invalid, so that is what I report. Three takes from one grader can't say which of us is right about training data, and I changed nothing after the grades came in.

Who it helps

Anyone building a motion dataset for a humanoid from human video: challenge entrants like the SuperSONIC field, research labs, and teams that collect demonstrations for a living. The one who pays is whoever pays for the fine-tune. A bad take costs GPU hours before anyone notices, and a list of seconds to film again costs a fraction of a cent.

How I built it

Fusion runs on Nebius Token Factory with nvidia/Nemotron-3_5-Lightning, and the second vision judge is nvidia/Nemotron-3.5-Super-VL on Token Factory too. Neither model touches the physics. The measurements and the thresholds are plain code, and the guard decides whether a model's answer is allowed to stand.

Thresholds first. config/thresholds.yaml went into git in 0c3a5ce before any graded run, and it's the only commit that has ever touched it. The cutoffs are engineering settings, not findings: mean joint error 0.5 rad, worst (95th percentile) 1.0 rad, a fall below 0.2 m root height, 35 degrees of tilt, 0.5 s out of balance, 0.5 s of foot slip. Any fall or any visual mismatch is a re-show.

A replay that can't quietly lie. Showhand precomputes every body frame, then publishes Protocol v3 frames at 50 Hz over ZMQ into SONIC behind a frozen 10 ms jitter gate. The run id has to match across the replay timing, the simulator metadata and the SONIC console, and SONIC's log has to contain every frame index in order. Rendering is a separate pass, so drawing can't slow control.

Cosmos on Nebius AI Cloud. nvidia/Cosmos-Reason1-7B, pinned to a fixed revision, runs in bfloat16 with greedy decoding in a Nebius AI Cloud Serverless Job on an L40S platform, in a PyTorch image pinned by digest. One clip's frame pairs were 290,162 bytes against the job's 65,536-byte limit for injected files, so the pairs go to a private Object Storage bucket the job mounts read-only. Loading the model took about 282 s each time. The validator only accepts a job that finished in the COMPLETED state.

Fusion on Token Factory. Lightning gets the metrics and the visual verdicts and has to answer a strict JSON schema at temperature 0, thinking off, 1,200 tokens max. If the guard refuses, Nemotron gets one retry that's told the refusal reason. If that's refused too, Showhand re-shows every failing second. I added the retry and the fallback after my first real takes and committed them as a dated amendment to the pre-registration, before any outside grade existed.

A second vision judge. I sent the same 20 windows and the same question to Super VL on Token Factory. All 20 answers were valid, and it called 19 of them a mismatch. It agreed with Cosmos on 3 of 20 and with the physics on 2 of 20, while Cosmos agreed with the physics on 15 of 20. That's why the decision rests on measurements first: two vision models looking at the same frames can disagree that much. Super VL shows on the replay as a second opinion and doesn't feed the decision. The whole pass cost $0.0047.

Pre-registered evaluation. config/residual_prereg.yaml fixes the comparison before real labels: Cohen's kappa against an outside human's accept or re-show grade, the fused judge against tracking error alone, a paired whole-take bootstrap with 10,000 replicates and seed 20261005. Synthetic rows are for unit tests only. The file hasn't changed since; the first real grades and the result it gives are in results/residual/.

The replay page. TypeScript, three.js and Vite, built only from the committed records. The G1 is drawn from MuJoCo's own per-body poses, and the phone video is the clock, so the robot, the timeline and the detail card all follow the same second. It's checked by 61 unit tests and 19 Playwright tests, including axe accessibility checks in light and dark mode, and I run the same Playwright suite against the live site.

Plumbing check first. Before my own takes I ran two licensed Pexels clips through the pipeline, labelled PLUMBING TEST. The simulated G1 fell 7 and 14 times. The visual job never started on them because the Nebius CLI asked for browser sign-in and timed out, so their records show the judge blocked and $0 of GPU, with nothing filled in.

Physical AI without a robot in the room

The hackathon FAQ says the Physical AI track has no required hardware and simulation is fine, and that's what this is: a real human body on a real phone camera, and a Unitree G1 model driven by NVIDIA's whole-body controller in MuJoCo physics. Falls, tilt and foot slip are physics measurements of that simulated robot. No real G1 ran, and I don't claim one did.

I tried live first. On my 8 GB card GEM-X averaged 2.0 to 2.5 frames per second, far from the 15 a live loop needs, and the full local run peaked at 7,922 of 8,192 MiB of GPU memory. So Showhand grades recorded takes, and the vision model runs as a Nebius job instead of on my desk.

Challenges I ran into

  • GEM-X setup. One run was killed in the image-feature path, one failed on a missing sam_3d_body module, and the retargeter's vendor files arrived as Git LFS pointer text ("could not convert string to float: 'size'").
  • WSL scheduling. Four long-clip replays broke the 10 ms jitter gate (worst 143 ms), one had a 342 ms telemetry gap, and one was thrown out because SONIC logged frames 5 to 880 instead of 0 to 880. WSL-native log storage, a 3 ms pacing margin and a publisher warmup fixed it without moving the gate. My takes' worst jitter was 0.10 ms.
  • The 65,536-byte job limit and the sign-in prompt above. Both stopped real runs before I worked around them.
  • Nemotron's first answer was refused on all three takes. That's why the told-the-reason retry and the fallback exist.
  • The vision models disagree. Super VL called 19 of 20 seconds a mismatch where Cosmos called 2. I kept the decision tied to the measurements and show both verdicts instead of picking the one I liked.
  • The first human disagreed with me. An outside grader kept all three takes Showhand sent back. Loosening the thresholds to match that grade would have been easy and would have made the pre-registration meaningless, so the thresholds stayed and the disagreement is in the report.

Accomplishments that I'm proud of

  • Three real takes graded end to end, each with exact seconds to show again and a record of how the decision was reached, for $0.00087 of fusion.
  • Thresholds frozen in git before any graded run, and a pre-registration amended with a date before any human grade, then scored as written when the first grade disagreed with me.
  • A replay page where anyone can scrub my phone video and the robot together and click into the evidence for any second.
  • When a stage didn't run, the record says so and shows $0 instead of a filled-in answer.

What I learned

  • Plumbing is most of a physical AI project. More of my commits prove a replay happened as recorded than touch a model.
  • Don't move the gate to pass. Every jitter failure was my setup, not the threshold.
  • Freeze the rules before the data, and if a procedure changes, date it before the labels arrive.
  • A guard checks that an answer is grounded, not that it's complete. take_0526's passed answer re-showed one failing second and skipped another.
  • Physics and vision see different failures. take_0527 had no failing second in the metrics, and its re-show came from Cosmos. Cosmos flagged orientation twice, and my metrics have no yaw term at all.
  • A human watching a 6-second clip is a different judge from a threshold. My grader let through half-second balance slips that the frozen limits catch. Whether those slips matter is the question the next takes have to answer.

What's next

More takes, so the pre-registered comparison can return a valid number. The grader asked for the move they care about, a Scorpion "get over here" into an uppercut, and offered to grade it.

Then: add a yaw term to the metrics and run the residual comparison on every new graded take.

So the person holding the phone knows which seconds to film again before the GPU hours go in.

Replay: https://showhand-replay.vercel.app

Code: https://github.com/StephenSook/showhand (Apache-2.0)

Built With

  • ffmpeg
  • mujoco
  • nebius-ai-cloud
  • nebius-object-storage
  • nebius-token-factory
  • nemotron
  • numpy
  • nvidia-cosmos-reason1
  • nvidia-gear-sonic
  • nvidia-gem-x
  • nvidia-nemotron-3.5-lightning
  • nvidia-nemotron-3.5-super-vl
  • opencv
  • playwright
  • python
  • pytorch
  • three.js
  • typescript
  • unitree-g1
  • vercel
  • vite
  • zeromq
Share this project:

Updates