RoBuff

Pinch now, grip 2.56 seconds later.

Comment

Inspiration

A 2024 University of Bristol study found that a robot arm run at delays up to 2.6 seconds got worse on every measure the authors recorded, and light alone takes 2.56 seconds to reach the Moon and come back. Germany's DLR flew predictive graphics on the ROTEX arm in 1993 to work around exactly this, but almost nobody gets to feel the problem. I wanted people to feel light delay with their own hand instead of reading about it.

What it does

RoBuff is a browser testbed where your bare hand, seen by a laptop webcam, drives a physics-simulated SO-ARM100 robot arm that picks things up by friction alone. Switch to Moon mode and your commands take 1.28 seconds to reach the arm while the view takes 1.28 seconds to come back; a see-through "ghost" arm, a second simulation that does not know the rock's true mass, answers at once. You can ask for objects such as "an asteroid the size of Texas", and a guardrail checks each one against the simulated arm's measured limits and a simulated lift test before it appears; a gravity menu switches between Earth, Mars, the Moon, the asteroid Bennu and orbit. Everything is a simulation, and the screen says so at all times.

You can also move the arm by voice, with phrases like "move left 5 cm", "pick it up" and "stop". Recognisable things like a car are built from coloured parts. Any object can wear a skin drawn by Nano Banana, Google's Gemini image models, from a short description. A skin changes only the look, never the physics.

How we built it

  • Hand input: MediaPipe's hand landmarker runs on device and gives 21 points per hand; a One Euro filter and a pinch latch (close below 0.25, open above 0.38, 2 frames) turn them into a palm target and a grip. Median 11 ms per frame in a fake-camera test, a hand in 123 of 123 frames (RTX 4050). Chosen because it runs in the browser with no camera frames leaving the laptop.
  • Physics: MuJoCo 4.0.0 (the official WebAssembly build) runs the public MuJoCo Menagerie SO-ARM100 model, because it simulates contact and friction, so a grip can slip. 240 frames per second headless.
  • Kinematics: our own damped least squares IK on MuJoCo's Jacobian; reachable test targets landed within 0.09 to 1.85 mm.
  • Guardrail and lift test: our own checks (mesh, convex hull, jaw direction, scale, mass, friction, grip height), then a headless pick on a second copy of the physics state; only a rise of at least 48 mm counts. Procedural meshes are generated in the browser.
  • Gemini: gemini-3.8-flash turns a request into an object spec under a JSON schema, through a small local key server, so the key never reaches the browser. Gemini proposes; code decides. Live check: 5 of 5 typed requests gave schema-valid specs, median 3867 ms end to end. Without a key, an offline keyword mock runs and is labelled.
  • Voice: hold V to talk, or press Talk live for hands-free talk; an offline grammar handles stop, home, gravity, delay and ghost; spoken receipts with captions, voiced by ElevenLabs (eleven_flash_v2_5, voice Roger) through the local key server; live check on Nov 7 at 21:42: 5 of 5 lines, first audio median 176 ms. Without a key the browser's own voice speaks, labelled offline.
  • Voice motion: the offline grammar turns a phrase into a direction and a distance. The move goes in on the same control path as the hand, so the stop, the reach limit, the Moon delay and the ghost apply to it. Looser requests go to Gemini 3.8 Flash, which can pick up to 3 actions from a fixed list. In a browser test "move left 5 cm" moved the gripper 49.4 mm, and a stop mid-move let it drift 0.4 mm.
  • Skins: the key server asks Nano Banana for a 1024x1024 texture. High quality uses gemini-3-pro-image and is the default. Fast uses gemini-3.1-flash-lite-image. Pictures are cached on disk and capped at 100 units a day, because each new one is a paid call. Without a key, the browser draws a procedural pattern and says so. Mass, friction and collision shapes never change.
  • Delay and ghost: delay lines on the commands and on the view (1.28 s each way, from the NASA NSSDC mean distance of 384,400 km), and a second compiled world for the ghost with a rock of a different mass and friction; an error meter in millimetres.
  • Trial logger: every grab logged to CSV with a fixed column list; a scorer with its pass rule written before any data.
  • Safety: Space (and, experimentally, two open hands) latches a stop that holds every joint and the jaw; only Resume on screen clears it.
  • Tests: Vitest unit tests and Playwright browser tests on Chrome, including a scripted synthetic hand and a fake camera. On the final commit, a fresh copy of the repository with no keys passed 381 unit tests in 46 files, 27 key server tests, and the pick, voice-motion and skins browser tests.
  • Built with TypeScript, Vite and three.js.

Challenges we ran into

  • Depth from a single webcam: how big the palm looks drives reach, which is noisy, so the arm target is smoothed and clamped to a reachable workspace.
  • Contact at very low gravity is not trustworthy in the solver: at Bennu gravity a 5 cm/s sideways push lifted the resting cube 46.9 mm. We say so on screen instead of hiding it.
  • Slippery parts: a shard of comet ice passed a weight-only friction check and still dropped in the lift test; the fix was to measure how square the pads can meet the hull (14 degrees off square, so it needs friction 0.26 where ice has 0.03).
  • The ghost meter barely moved: with the mass and friction mismatch we fixed before testing, it never left green in synthetic runs (max 1.19 mm). We did not change the mismatch to make it move.
  • No API keys until the evening, so every key-dependent part had to work offline first and be labelled.

Accomplishments that we're proud of

  • Scripted friction pick: rose 78.2 mm and held 5.65 s; 5 of 5 repeats from the same start (deterministic, not proof of robustness); 6 of 9 cube positions passed, the 3 failures all 8 to 12 cm off the arm's centre line.
  • Fake-camera hand tracking: a hand in 123 of 123 frames, median 11 ms.
  • Lift verdicts in the browser: Bennu-like sample passed (rise 77.3 mm, held 2.02 s); "an asteroid the size of Texas" rescaled to a 55 mm model of 144 g that lifted 76.0 mm; "solid iron, 60 mm" refused at 562 g; comet ice refused (needs 0.26, has 0.03).
  • Gravity drops match theory: 0.354 s on the Moon (theory 0.351 s), 0.144 s on Earth (theory 0.143 s).
  • Gemini live check: 5 of 5 schema-valid specs, median 3867 ms.
  • Voice motion: "pick it up" lifted the test cube 78.5 mm with both jaws on it.
  • Ghost gate: not yet measured (0 logged trials).

What we learned

Without help, people move and wait: Ferrell described it in 1965, and the cost grows with the delay. The literature says prediction helps reaching more than contact, and our own meter showed how small a fixed mass mismatch looks during a firm grip. And a simulation is only honest if it says what it cannot do.

What's next for RoBuff

More operators than one, more than one object at a time, a second camera view for depth, and a test on real hardware.

Try it

It runs locally: see the README's "Try it" section in the repository at https://github.com/ahmedsohail2003/RoBuff. The project page is at https://ahmedsohail2003.github.io/RoBuff/.

Disclosure

  • New: all application code, written on November 7 and 8, 2026 from an empty repo; the first commit is at 1:43 PM EST, after hacking opened at 1:30 PM.
  • Existed before: the public SO-ARM100 arm model, the libraries below, the public hand-pose model, and NASA and JPL data.
  • Simulated: both arms, the light delay, the gravity and the contact. No real robot is connected.
  • Mocked: object specs from an offline keyword mock and the browser voice whenever no key is present; procedural skins when Nano Banana is not available; cached voice lines when replayed. All labelled on screen.
  • Unfinished or untested: the ghost trials (0 logged); a real-hand try of the two-hands stop gesture (not logged); voice motion with a real microphone (the voice tests used typed-in transcripts). Real-hand control was tried by the author on a laptop webcam and worked, but it was not timed and no numbers were logged, so every measured number above comes from scripted, photo and fake-camera tests.
  • AI tools: Claude Code (Anthropic) for planning, research summaries, code, tests and docs, under the author's direction and review; Gemini (including Nano Banana for skins) and ElevenLabs inside the product as described above. Skins are AI-made images and carry Google's SynthID watermark.
  • Related earlier projects by the author: WorkCell, EmberHacks "The cell" (submitted to another event), HandSignal and FleetLine. None of their code is used here. EmberHacks also had voice commands for its robot and a Gemini image model that printed labels on parts. RoBuff's voice motion and skins were written from scratch during this hackathon and work differently.
  • Privacy: no camera frames leave the laptop; Gemini receives only text (the request, and for Talk live the recent turns and the app's state); Nano Banana receives only the skin description.

Credits

Libraries, models and licences (full list in THIRD_PARTY.md): MuJoCo 4.0.0 (Apache-2.0); MuJoCo Menagerie trs_so_arm100, pinned at commit 98a70c5 (Apache-2.0), derived from The Robot Studio's SO-ARM100; MediaPipe Tasks Vision 1.2.2 and the hand landmarker model (Apache-2.0); three.js 0.187.0 (MIT); Vite, Vitest (MIT); Playwright, TypeScript (Apache-2.0).

Sources, all retrieved 2026-11-07: NASA NSSDC Moon, Earth and Mars fact sheets (distance and gravity); JPL Small-Body Database, Bennu (GM 4.8904 m³/s², diameter 484.44 m); Louca et al. 2024, ACM Transactions on Human-Robot Interaction, doi:10.1145/3651993; Ferrell 1965, IEEE Transactions on Human Factors in Electronics, doi:10.1109/THFE.1965.6591253; Sheridan 1984, NTRS 19850006202; Richter et al. 2019, IEEE ICRA, arXiv 1809.08627; DLR's ROTEX history page; Bennu returned-sample density from OSIRIS-REx (JAMS 2025 abstract R5-13); the Canadian Space Agency's Canadarm3 page; Engineering ToolBox friction coefficients.

Built With

Share this project:

Updates

Submission history