weird flex but ok

web ui

Inspiration

What if machines could learn from our motion and force, and then extend our body far beyond human limits? We started with that vision — combining gesture and muscle signals to control machines. While pose tracking can handle position, EMG sensors are agnostic to lighting and offer a robust way to capture force and intent. This hybrid input system opens the door to more precise, expressive, and adaptable control.

What it does

Tracks your shoulder, elbow, wrist, and wrist roll using only a webcam - no depth camera, no wearables. Uses an EMG sensor to detect myoelectric signals from your forearm and control the grip strength/hand closing of the robot hand. Translates poses and force control into real-time robotic arm movement Use voice input to adjust the parameters- e.g., say move 10x faster/harder than how i do it, use Fish Audio and feed the parameters to change the robotic movement- human enhancement. (WIP).

How we built it

Input: Webcam + BlazePose for 3D keypoints from shoulder to wrist, MediaPipe Hands for palm roll, EMG sensor on the forearm to detect muscle flex and map to robotic grip. JavaScript for inference and a WebSocket bridge from the browser to Python for robotic control Converted (x, y, z) positions into joint angles + motor instructions for a robotic arm, filtering & smoothing applied to noisy pose estimates Used Phospho for reactive signal mapping and logic control

Challenges we ran into

Wrist roll was unstable with BlazePose alone — required fusion with MediaPipe Hands Webcam pose is jumpy without a depth camera — needed filtering to reduce jitter EMG signal interpretation is nonlinear — requires thresholding and calibration The robot arm’s default motor config was dangerous (too fast) — tuning acceleration was essential Hardware issues: fragile joints, over-rotation, screws falling out mid-test

Accomplishments we’re proud of

Created a full gesture + force → robot control pipeline using just open-source tools Got the robot to move smoothly in real time, with no controller hardware Successfully mapped EMG flex signal to robotic grip Integrated Phospho as a control framework, learned (and debugged) a lot about low-level motion control and filtering Work with a beautiful team and made amazing friends :)

What we learned

Combining multiple imperfect sensors (pose + EMG) can lead to a stronger interface Smoothing and constraint logic are essential for real-world robot safety A simple, intuitive interface beats a complex one — motion is a language Hardware always breaks. Design for resilience.

What’s next

Gesture memory: record, store, and replay physical demos Condition-based automation: trigger actions based on external cues (e.g. voice command) Human++ mode: replay motions 100x faster or with amplified strength Multi-arm or multi-agent control: control multiple arms or teammates via gestures Fine motor control via EMG waveform analysis