Inspiration

Every device around us has its own remote, app or control panel. Before you can do anything, you have to find the right one. We wanted something seamless: look at a device, move your hand, and it responds. No phone, no clicker, no app.

What it does

Midi-chlorians is a universal remote you wear. It combines a Vexo wristband with Mentra Live smart glasses.

  • Look to select. The glasses work out which device you're facing.
  • Gesture to control. Swipe your wrist left or right to step, tap the band to toggle, double-tap to select.
  • Speak for everything else. Say "Vexo" followed by a command, and an agent turns your words into an action on the selected device.

How we built it

  • Band. A web page connects to the Vexo band over Web Bluetooth and reads the motion sensor at 100 samples a second. Our gesture detector turns that stream into three events: step, toggle and select.
  • Voice. The band's microphone streams to Deepgram. We listen for the wake word "Vexo" and send what follows as a command.
  • Device API. Every input posts to one small API with a strict contract. Each device sits behind an adapter that understands the same few actions, so adding a device means writing one adapter.

Challenges we ran into

  • Gesture controls It sits above the wrist joint, so a hand flick mostly registers as forearm roll. Our twist and flick gestures were indistinguishable. We cut twist and rebuilt around left and right, which the sensor can actually tell apart.
  • Raise-to-talk didn't work reliably, so we switched to a wake word.

What we learned

  • Three gestures that work every time beat five that work most of the time.
  • When a device can't talk back, the action itself has to be the feedback.

What's next

  • More devices: lights, speakers and TVs through one smart-home adapter.
  • Testing with people who have limited hand mobility, who stand to gain the most from one small set of large gestures.
  • Recognizing devices without markers or setup.

Built With

Share this project:

Updates

Submission history