STARNOSE

Inspiration

EOD robots are teleoperated over radio. Two failure modes come up in the literature and in conversations with EOD specialists through my work at the Wicked Problems Lab, Vanderbilt Institute of National Security: the operator cannot feel what the gripper is doing, and the link degrades or drops with the arm mid-task. Nobody asked for more autonomy. STARNOSE is software for those two problems on a low-cost arm, with the operator keeping authority.

What it does

Runs on an SO-101 arm pair (leader and follower) with inert props. Operator-driven.

• Contact sensing: a microphone in the gripper plus servo load through a C++17 DSP front end (FFT, onset detection). Contact and load are shown on the operator's phone in real time.
• Step checker: Nemotron 3.5 Super VL on Nebius Token Factory reads the camera frame and the contact events about once per second and returns one token, continue or pause, with logprobs. It can pause the arm. It cannot start or resume it.
• Link awareness: the phone echoes heartbeats; a Gilbert-Elliott filter and a NORMAL, DEGRADED, LOST state machine on the Mac classify the link. On LOST the arm runs the preset the operator chose before the mission: HOLD (freeze) or STOW (move to a stowed posture at a capped rate).
• Safety supervisor: C++17, called on every command. Rejects non-finite values, holds the measured position, clamps joint limits, caps the per-tick step, logs overrides.
• GR00T N1.7 audio tokens: an audio-token module adds gated contact-audio embeddings (frozen EfficientAT) as tokens beside state and action in the GR00T N1.7 DiT. At initialization the patched policy equals the base policy bit for bit.
• Operator console: web page and a native iOS app (SwiftUI). The app adds Core Haptics taps on contact onsets and a hand-tracking teleop mode through Apple Vision. The app builds; it has not yet been run on a phone against the arm.

No learned policy has driven the arm.

How we built it

• One C++17 library (DSP, filter, state machine, supervisor), pybind11 bindings, GoogleTest under ASan and UBSan: 74 tests.
• LeRobot 0.6.1 robot type starnose_so101 whose send_action always goes through the supervisor. A starnose_phone teleoperator for the hand-tracking mode.
• Console hub: aiohttp event bus on one port, topics for joints, audio, link, checker, cmd, hand, forcepad. Python suite: 616 tests.
• Nebius AI Cloud: scripts/vm/run_all.sh creates an L40S VM, installs LeRobot, caches the GR00T weights and starts training in tmux. 2026-10-30: condition A (vision + state) on the public lerobot/svla_so101_pickplace dataset, 600 steps, loss 1.15 to 0.18 at 0.54 s per step; condition B (audio tokens) on the same data with synthetic contact features, loss 1.16 to 0.18. The B run proves the audio path trains on the GPU; the synthetic features carry no information, so it says nothing about the audio hypothesis. A 3000 step baseline run followed (loss 0.09 at step 3000). No pouch demonstrations were recorded, so no pouch-task number is claimed.
• Nebius Token Factory: the checker (Nemotron 3.5 Super VL) and the after-action report (Nemotron 3 Super 120B). One 72 minute real-arm session produced 1,735 checker calls.
• MuJoCo scene of the SO-101 for the link experiment and the B-roll, with the C++ supervisor in the loop.
• ESP32 force-pad firmware and a host reader are written; the firmware is not flashed.

Challenges we ran into

• LeRobot 0.6.1's async server cannot serve a GR00T checkpoint trained with relative actions: it decodes one step at a time and the decode step refuses. Fixed in our vendored server (decode the whole chunk); 44 serving tests pass, and both checkpoints served on the L40S: median round trip 132 ms (baseline) and 163 ms (audio tokens) for 16-action chunks.
• The live checker crashed and reconnected on every link message with a null RTT (before the first echo). Fixed; 22 checker tests pass.
• The Nebius CLI login is browser-only and its OAuth callback port changes per attempt. Over SSH, the callback URL had to be relayed by hand with curl to the Mac.
• macOS tar added AppleDouble ._ files to a dataset; LeRobot read ._file-000.parquet as Parquet and failed. Deleted on arrival.
• The checker threshold (0.50) is uncalibrated: 316 pauses in 72 minutes. Calibration needs hand-labelled frames, which were not made.

Accomplishments that we're proud of

• Real arm, 72 minute teleop session through the supervisor: 45,930 joint frames, 1,735 live Nemotron VL checks. When the phone went offline the estimator went DEGRADED and LOST 0.3 s later, the arm held, and P(pause) reached 1.00.
• Supervisor property test: 1,000,000 random actions, zero outputs outside limits, non-finite, or over the step cap.
• Link filter in simulation, 4G Rural tier: 30 of 110 bad stretches detected at a median 350 ms, against 8 of 116 at 625 ms for a 500 ms timeout.
• Three GR00T N1.7 fine-tunes on Nebius, with and without audio tokens, from one command, and both checkpoints served.

What we learned

• Decode relative-action chunks whole; the per-step path in LeRobot 0.6.1 is wrong for GR00T.
• A one-token answer with logprobs turns a VL model into a usable probability at 1 Hz.
• A checker that can only pause fails safe, but an uncalibrated threshold costs operator attention.
• Format every nullable field defensively; one null took a service down.

What's next

• Record the pouch demonstrations and run the pre-registered A/B/C trials (15 per condition) with the audio tokens on real contact sound.
• Calibrate the checker on hand-labelled frames.
• Flash the force pads. Run the iOS app and hand teleop on the arm.
• Prior work: ListenVLA (arXiv 2610.16313) put external-sound tokens into GR00T N1.7. Ours is contact sound, open source, on a low-cost arm.

Built With

• c++17