Fetch

Inspiration

Asking a robot for something still means pulling out your phone, saying a command out loud, or grabbing a controller. All of those take your hands, your eyes, or your voice. We wanted something quieter: what if you could just write a word in the air, flick your fingers like Spider-Man shooting a web, and have the object land in your hand?

What it does

Write: trace a word like "pen" or "screwdriver" with your finger. Nothing is actually written: a neural wristband reads the word from the electrical activity of your forearm muscles (EMG).
Flick: make the Spiderman gesture (middle and ring fingers curled, index, pinky and thumb extended) to confirm the command, so the robot never acts on an accidental movement.
Fetch: a camera and Grounding DINO find the object that best matches the word, and an SO-101 robot arm grabs it and drops it in your hand or places it at a set spot.

If an object is outside one arm's reach, two arms work together: the first picks it up and hands it to the second, which delivers it.

How we built it

Sensing. We stream raw EMG from a Meta neural band through Kinesis-Axo: 8 channels at 2,048 Hz per channel, in batches of 16 samples, plus a 128 Hz gyroscope.

Labeled data. To train decoders, we built a capture studio (React + MediaPipe in the browser, with a Python recorder on the same Mac). A camera tracks the hand while the band records, so every EMG window gets a label from what the hand is actually doing. For example, grasp strength is the mean of 14 finger-flexion angles, normalized between each wearer's open-hand and fist calibration:

$$ g = \operatorname{clip}!\left(\frac{\bar{\theta} - \bar{\theta}{\text{open}}}{\bar{\theta}{\text{fist}} - \bar{\theta}_{\text{open}}},\ 0,\ 1\right) $$

Decoding. Word recognition comes built into AXO, which turns the band’s signals into words like “pen” and “toy”. The two gestures that control the robot are ours: we collected our own labeled EMG data with the capture studio and trained models to detect the Spiderman gesture (which sends the command) and the grasp (which tells the arm to drop the object into your hand).

Grounding and grasping. The decoded word becomes a text prompt for Grounding DINO, an open-vocabulary detector, so the arms can find objects they were never specifically trained on. [Describe how a detection becomes a grasp: camera calibration, choosing the grasp point, controlling the SO-101.] The overhead camera position is fixed overlooking the table, and by calibrating the camera and the robot arms together, a deterministic mapping can be made from pixels in the camera frame to 3d coordinates. When an object is detected this mapping is used to calculate the desired 3d coordinate end effector position to pick up the object. Inverse kinematics is then used to derive corresponding robot joint angles to achieve this end effector pose. The robot follows these calculated joint angles to reach and pick up the object.

mediapipe is then used to track hands that enter the frame, and through a similar methodology the robot arm travels to and drops the object in an open hand.

Two-arm handoff. Whenever the target drop-off position for an object is outside the reach of one arm but within reach for the other, the arms coordinate a handoff. During this handoff the object is transported to a middle-ground that is accessible by both arms where it is then picked up by the arm that is capable of reaching the target drop-off position.

haptic-feedback When the robot has arrived at a hand and is ready to drop the object, it does not immediately do so. Using the streamed EMG data a closed fist instructs the robot to keep holding onto the object, while a opened hand instructs the robot to release the object and drop it into the hand. If the robot is either waiting for the hand to open or the hand is out of reach, it will send a haptic buzz through the neural band to inform the user.

Challenges we ran into

Synchronizing camera and muscle signals. The band, the camera and the browser each keep their own time. We align them in software on the Mac's monotonic clock using repeated round-trip probes, and refuse to record unless the sync is fresh and tight (round trip under 50 ms). Bluetooth and camera delays remain unknown, so we keep all raw timestamps for offline correction.
Noisy, personal signals. EMG varies between people, between sessions, and with how the band sits on the arm, so calibration matters a lot.
Imperfect ground truth. Camera hand-tracking struggles with tight fists and hidden fingers, so our labels are geometric approximations, not perfect anatomy.
The handoff. [Add what was hard about getting two arms to pass objects reliably.]

Accomplishments that we're proud of

Training our own EMG detectors for the Spiderman and grasp gestures from data we collected ourselves
Turning a word written in the air into a real object in your hand
Two arms handing objects to each other to cover a larger space

What we learned

How much of a muscle-signal pipeline is really a data and timing problem: a decoder is only as good as its labels and how well they line up
Open-vocabulary vision models like Grounding DINO make "fetch anything" realistic without per-object training
[Anything about robot control, or about teamwork under a 24-hour deadline]

What's next for Fetch

Fetch was a 24-hour build, but it points at something bigger: silent, hands-free control for people who can't easily speak, reach, or use a screen. A larger vocabulary, more commands (like placing a watch on its charger), and calibration that carries over between users would make it practical outside a hackathon table.