Tether

Tether estimates the physics gap of a pushing task from video or logs, shows where measurements are missing, and provides the next experiment and a verifiable setup.

Comment

Inspiration

A controller can work in simulation and miss on a different surface because friction, actuator strength or the camera is wrong. Engineers usually close that gap by watching failures, changing parameters and retraining. Tether turns those measurements into a simulator correction, with evidence for what was measured and an explicit unknown where data is missing.

Robot and factory pushing are the lead cases. A driving example applies the same simplified sliding model at another scale; it is not a validation of real vehicle dynamics.

What it does

Studio follows three steps: Choose data -> What we found -> Use it.

  1. Start with a video, robot log or hidden world generated in NVIDIA Newton. Examples are tracked and analyzed automatically; uploaded videos first let you check the reference rectangle.
  2. Read a plain-language result, before/after predictions and the next experiment. Open separate views for measurement, the AI notebook, parameter intervals and retraining.
  3. Download a configuration or report for NVIDIA Newton, Isaac Lab or CARLA, plus JSON. The Newton check distinguishes withheld-push prediction from a replay of data used for fitting.

A Benchmark page tests the agent on worlds whose physics it cannot see. Ask Tether answers questions grounded in the current session, in English or Korean.

How we built it

NVIDIA Nemotron 3 Super on Nebius Token Factory is a tool-using agent. It examines deceleration, checks perception, proposes model structures, tests hypotheses and requests informative experiments. The agent chooses structures and free variables; SciPy least squares computes the numbers. A library search cross-checks the result and rejects unsupported explanations. Without attached hardware, experiment requests become queued cards rather than physical actions.

NVIDIA Newton and Warp, running on CPU, render synthetic videos, retrain the same learner in 16 parallel worlds and replay exported JSON in a separate process. With at least six usable pushes, roughly the first 80% calibrates and the remainder evaluates; otherwise the result is labeled fit replay. Bootstrap intervals account for measurement uncertainty, and a single usable push has no interval.

The UI and FastAPI service run on Google Cloud Run. Runtime model inference goes to Nebius Token Factory; no Nebius GPU VM was provisioned. The calibration agent can also run through NVIDIA NeMo Agent Toolkit for workflow tracing. NVIDIA Cosmos Reason 2, run locally, is an optional second opinion on sliding versus tipping. MuJoCo supplies an independent simulation engine for evaluation.

Results and what they mean

  • Gap-Bench, NVIDIA Newton, 15 worlds per tier: on compound shifts, mean hidden-world success was 22% for the nominal simulator, 54% for rules, 96% for the agent and 97% for system identification. The agent did not beat the strong system-identification baseline.
  • Public real objects, EV-RealPhys: friction was within 0.05 of the independent tilt test for 4 of 5 objects on both log and video paths. Mean absolute friction error was 0.019 for logs and 0.029 for video. Same-input quadratic median baselines were 0.018 and 0.037, respectively. These small-sample results do not establish an accuracy advantage.
  • 35-case Studio comparison: 24 MuJoCo worlds, six Newton examples and five public-real logs, with the final approximately 20% withheld before model calls. Mean case-median stop error was 3.027 cm for offline structure search and 3.018 cm for the agent, with or without cross-checking. The paired difference was -0.0096 cm; its 95% case-bootstrap interval includes zero. All cases completed, with no cross-check overrules.
  • Local verification: 148 offline tests passed on October 18, 2026. Recorded model responses replay without paid API calls.

The contribution is a flexible tool-driven investigation, model review, explanations and next-experiment guidance. We do not claim an accuracy advantage over a well-designed numerical baseline.

Challenges and lessons

Asking the language model to tune numbers directly did not converge. Separating structure selection from numerical fitting fixed that. A seam between two simulated table bodies tipped sliding objects; a per-body Warp friction-switching kernel removed the seam. Bootstrap intervals initially ignored errors shared across runs, so camera scale and speed uncertainty now enter the resampling.

The largest lesson was to test the explanation, not just the final stop. Friction and actuator gain can trade off while producing similar stops; launch residuals and perception checks help expose that ambiguity. More data does not automatically remove shared measurement bias.

Evidence and limits

Application-flow examples are Newton-rendered synthetic data. Real-object evidence comes from public EV-RealPhys, using dataset camera calibration and motion-capture-assisted tracking inputs: 25 of 45 video tracks were retained and 20 excluded. Independent phone-and-A4 validation remains unperformed and is deferred until after submission.

The reliability indicator is a diagnostic policy, not guaranteed interval coverage. Real-data intervals remain too narrow. Isaac Lab and CARLA exports are format checked against stand-ins, not executed in those engines. User comprehension has not been tested. The physics covers launched sliding to a stop, not grasping, steering or lane changes.

What's next

Collect independent phone recordings with a reference rectangle and separate tilt measurements, evaluate held-out pushes, and run a small first-time-user study. Further engine integration comes after that evidence, not before it.

Try it and read more

Real-object footage: EV-RealPhys (Kandukuri, Strecke, Stueckler, 2023, Max Planck Institute), CC BY-SA 4.0. Tether is an independent hackathon project, not affiliated with or endorsed by NVIDIA or Nebius.

Built With

Share this project:

Updates

Submission history