MindArm

An assistive robot arm you command by choosing goals, not joints: NVIDIA Nemotron on Nebius Token Factory plans each pick, a safety gate checks it, and a Jetson Nano executes it.

Comment

Inspiration

People living with ALS, high spinal cord injury, or locked-in syndrome often cannot use a joystick or a touchscreen. Non-invasive brain-computer interfaces can give them a channel, but a narrow one: a few reliable choices at a time. Asking that narrow channel to steer six joints in real time is slow and tiring. We wanted to flip it: let the brain choose the goal, and let a reasoning model turn that choice into a safe, short plan for a real arm.

What it does

MindArm is designed to be brain-controlled through intent selection. Its first hands-free input is a double blink captured with electrodes, and brain-signal decoding is the next stage. It runs a closed loop between a person, NVIDIA Nemotron, and a physical 6-DOF arm:

  1. The user engages the system.
  2. Nemotron 3 Super, served on Nebius Token Factory, reads the scene as text and proposes exactly four options, for example "Pick up the sponge" or "Home".
  3. The user selects one with the keyboard, or with an optional double blink that selects the configured object. In the planned BCI, the options are tiles flickering at 8.57, 10, 12, and 15 Hz, and looking at one selects it (SSVEP).
  4. Nemotron plans the motion using only a closed vocabulary: MOVE_ABOVE, GRASP, LIFT, MOVE_TO_POSE handover, RELEASE, HOME. It never outputs angles, pulses, or speeds.
  5. Token Factory decodes the plan against a strict JSON Schema built for the current scene. A local safety gate then checks it again: vocabulary, objects in the scene, the Home rule, joint limits, and speeds. Invalid output gets one repair prompt; a second failure falls back to HOME.
  6. A Jetson Nano executes the plan on the 3D-printed arm, ramping every servo within its own hard envelope. The user can cancel at any moment, and the arm returns HOME.

Where it stands today: the arm and the Nemotron planning are real and run live. The full chain ran on the real arm in 15 live sessions, each completing its five-step plan, and the best runs lifted the sponge and carried it to the handover pose. Selection uses the keyboard by default, or the optional double-blink detector shown in the demo, on one AD8232 channel read at 250 Hz through an ADS1115 on the Jetson. A blink is eye movement captured by electrodes, not brain activity. In a cued test the detector caught 8 of 10 double blinks, with some false detections still to reduce. Poses are taught by hand, the scene is given as text, and reliable grasping is the main open item. MindArm is a supervised tabletop prototype, not a medical device, and it has no clinical validation.

How we built it

  • Contract-first architecture. Every module (EEG, decoder, perception, planner, arm) is a Python Protocol with Pydantic v2 models that reject extra fields, coercion, NaN, and infinity. The orchestrator is a state machine (IDLE → PROPOSING → SELECTING → PLANNING → EXECUTING) that keeps polling input while perception, planning, and arm motion run on a worker. Fakes were replaced one module at a time, so the full loop ran end to end from day one.
  • Nemotron on Token Factory. Two calls per cycle through the OpenAI-compatible API:
    • Strict structured output (json_schema, strict: true) with one anyOf branch per action. The object label is an enum of the objects in the current scene.
    • Local validation as a second layer, one repair attempt that includes the validation error, and a deterministic fallback.
    • One retry for timeouts and HTTP 5xx, separate from the repair, so an operation makes at most three calls.
    • Reasoning set per operation. Every call is logged with its latency, tokens (cached and reasoning), and settings.
  • Measured, not assumed. python -m mindarm.bench runs the live planner on the fixture scene and plans the options the model just proposed, as the real loop does. It reports p50/p95, first-try validity, repairs, fallbacks, rejection reasons, cached tokens, and cost at Token Factory's own listed prices.
    • Before: JSON mode with reasoning on. Plans were valid on the first try 6/10, at 4.88 s p50 and 6.05 s p95, and every rejection was an object label on the wrong action.
    • After: strict schema with reasoning off, N=30 per operation. Proposals took 0.76 s p50 and 1.17 s p95. Plans took 0.54 s p50 and 0.99 s p95, and 30/30 were valid on the first try and complete (the five-step handover, or [HOME]).
    • A full cycle costs about $0.00027, with 67% of prompt tokens cached.
    • We chose the model with data. Ultra matched Super on validity at 5.6× the cost per cycle; Nano was cheapest but needed repairs and was the slowest. MindArm uses Nemotron 3 Super for both calls.
  • The cloud reasons, the edge enforces. Nemotron 3 Super (120B) runs on Token Factory. The Jetson Nano doesn't host it. Instead it does what an edge device can guarantee: bounded, deterministic servo control and an independent safety envelope that holds even if the laptop or the model misbehaves.
  • Real arm on Jetson Nano. The laptop preflights every target against limits and speeds in microseconds, then streams ramped moves over the SSH exec channel to a Python 3.6 service. The service drives a PCA9685 and enforces its own envelope. We kept the Jetson's firewall and disabled port forwarding, so the whole protocol runs over SSH stdin/stdout.
  • No position feedback, handled explicitly. At startup the service energizes the servos one at a time at HOME. A lost link stops the ramp and holds the posture. Cancel stops the move mid-ramp before HOME.
  • Pose teaching tool. A local web page streams the Jetson's USB camera and has a slider per joint. We used it to teach the pick, lift, and handover poses for a sponge at a taped spot, and the tool checks every saved pose against limits and speeds.
  • Double-blink detector. A deterministic, numpy-only rule on the laptop: 60/120 Hz notch, 1 s moving-mean baseline, 40 ms smoothing, and two positive peaks close together. Nemotron never sees the signal or the decision. The gesture selects one configured object and is ignored outside the selection step.
  • Live operator console. A local web page shows the signal, both cameras, the four options, the plan, and each joint while the real loop runs.
  • 120 offline tests with the network blocked: contract edge cases, retries and fallbacks, cancellation mid-motion, link loss, the Jetson service against a fake I2C bus, and the real CLI in a subprocess. A separate opt-in test calls the live API and fails if anything needed the fallback.

Challenges we ran into

  • Getting access at all. Peru was missing from Token Factory's billing country list, which blocked API key creation at first and cost us time close to the deadline.
  • Servos mounted off-center. The horns were attached without centering the servos, so every joint was calibrated by hand, joint by joint, with a test tool that ramps and releases safely.
  • A humming shoulder. At the vertical "candle" pose the shoulder sits at its balance point, and gear backlash made it hunt and buzz. Moving HOME 50 µs off balance silenced it.
  • No feedback means jumps. A released hobby servo jumps at full speed to its first pulse, so we designed a staggered, channel-by-channel startup.
  • A hardened Jetson. The firewall was on and SSH port forwarding was disabled. Instead of weakening them, we built a line-delimited JSON protocol over SSH stdio for the arm, the camera stream, and live EEG.
  • The model was occasionally creative. In the baseline benchmark, 4 of 10 plans broke the contract on the first try, all by attaching an object label to GRASP, LIFT, or RELEASE. The gate rejected each one and one repair prompt fixed it. A per-scene schema and an explicit handover sequence in the prompt brought first-try validity to 30/30.
  • Valid is not the same as complete. A live plan for "Pick cup" stopped at LIFT. It was valid, but the HOME that followed would have dropped the cup. Our first benchmark had missed this because it planned fixed goals instead of the model's own options. Now it plans the model's options and counts complete plans.
  • The last centimeter. With no position feedback or force sensing, the grasp depends on taught poses and a sponge on a taped spot. In early runs the jaws landed on one end of the sponge and rose empty. It took several tries, adjusting the sponge's placement, before the arm lifted it and carried it on camera.
  • Heart-monitor amplifiers on a forehead. The AD8232 was built for ECG, not EEG. Alpha did not measurably rise with eyes closed at the forehead, so we moved to a gesture we could measure. Spontaneous blinks can look like commands, which led us to a double-blink rule.
  • Electrical safety for EEG. Electrodes on a person call for care with the electrical setup, so MindArm stays a single-operator bench prototype until the EEG front end has galvanic isolation.

Accomplishments that we're proud of

  • A physical arm executing plans from Nemotron, where the model can only choose and never command a motor.
  • One continuous take: a double blink selects the sponge, Nemotron plans the motion, and the arm lifts the sponge and carries it to the handover pose.
  • Five safety checks between the language model and the servos.
  • A safety gate that catches real model mistakes. In the baseline it rejected 4 of 10 plans on the first try, and with Nano it caught 3 of 10 that broke the Home rule, which our schema doesn't cover. The repair loop fixed all of them before execution.
  • A reproducible benchmark that cut the model time per cycle from about 7 s to 1.3 s and the cost from $0.0011 to $0.00027, with every plan valid and complete in 30 runs.
  • A one-person build covering the electronics, the Jetson service, the planner, and the tests.

What we learned

  • Language models are most useful in a robot when they choose and plan inside a closed contract, not when they generate control signals.
  • Contract design beat model size. Once the prompt carried a per-scene schema and the exact handover sequence, Super without reasoning matched Ultra's validity at under a fifth of the cost. Reasoning added 1.2 s per plan and no validity.
  • Measure the loop you actually run. Planning fixed goals hid incomplete plans that the model's own options exposed.
  • In a test on a synthetic image, Nemotron 3.5 Super VL returned a valid scene description in 2.2 s, so camera perception looks feasible in the same loop.
  • Measure a signal before building on it. Our console's "alpha ratio" looked promising until a reanalysis showed it was eye movement dropping, not alpha rising.
  • For a narrow-bandwidth BCI, the hard part isn't the AI. It's designing choices a person can make reliably.

Potential impact

People who cannot use their hands, and who may have only a few reliable inputs, still want to do small things at a table for themselves. If the grasp and the input become dependable, the same loop could pick up an object and move it to a set spot with fewer requests to a caregiver. These are goals for future user testing, not results yet.

Four things make that path realistic:

  • Cheap to run. A full cycle costs about $0.00027 in model calls, so 100 cycles a day would cost under 3 cents. The arm itself is a 3D-printed body, hobby servos, and a Jetson Nano.
  • The input is replaceable. The decoder is one module behind a Python Protocol, and the loop only needs a few reliable choices. The keyboard and the double blink already plug in there; a head switch, an eye tracker, or an SSVEP decoder could do the same.
  • The pattern travels. A model that chooses inside a closed contract, with limits enforced on the device, fits other low-cost robots where a language model must never command a motor.
  • Open code. The MIT-licensed repository gives researchers a low-cost bench for testing intent selection on real hardware.

The path there runs through a repeatable grasp, camera perception, galvanic isolation, and sessions with the people it is meant for.

What's next for MindArm

  1. Perception from the Jetson's camera with Nemotron 3.5 Super VL, so the options match what is actually on the table.
  2. A repeatable grasp, with a measured success rate.
  3. Galvanic isolation for the EEG front end, a held-out blink test, then SSVEP tiles and brain-signal decoding validated on recordings.
  4. Object IDs, collision checks, and a hardware emergency stop before any use beyond a supervised table.

Prior work

Maha Nawaz, Abdul Basit, and Muhammad Shafique, "MindArm: Mechanized Intelligent Non-Invasive Neuro-Driven Prosthetic Arm System" (2024), describes a DNN-based EEG system with three fixed actions. This project selects goals with a voluntary double blink today and plans SSVEP intent selection next, instead of a DNN on raw EEG. It also uses scene-aware options from Nemotron rather than three fixed actions. The 3D-printed arm was designed for this hackathon, and all software was built during the submission period.

Built With

  • 3d-printing
  • ad8232
  • ads1115
  • jetson-nano
  • nebius-token-factory
  • numpy
  • nvidia-nemotron
  • openai-python
  • pca9685
  • pydantic
  • python
  • pyyaml
  • ssh
Share this project:

Updates

Submission history