Inspiration
Every device around us has its own remote, app or control panel. Before you can do anything, you have to find the right one. We wanted something seamless: look at a device, move your hand, and it responds. No phone, no clicker, no app.
What it does
Midi-chlorians is a universal remote you wear. It combines a Vexo wristband with Mentra Live smart glasses.
- Look to select. The glasses work out which device you're facing.
- Gesture to control. Swipe your wrist left or right to step, tap the band to toggle, double-tap to select.
- Speak for everything else. Say "Vexo" followed by a command, and an agent turns your words into an action on the selected device.
How we built it
- Band. A web page connects to the Vexo band over Web Bluetooth and reads the motion sensor at 100 samples a second. Our gesture detector turns that stream into three events:
step,toggleandselect. - Voice. The band's microphone streams to Deepgram. We listen for the wake word "Vexo" and send what follows as a
command. - Device API. Every input posts to one small API with a strict contract. Each device sits behind an adapter that understands the same few actions, so adding a device means writing one adapter.
Challenges we ran into
- Gesture controls It sits above the wrist joint, so a hand flick mostly registers as forearm roll. Our twist and flick gestures were indistinguishable. We cut twist and rebuilt around left and right, which the sensor can actually tell apart.
- Raise-to-talk didn't work reliably, so we switched to a wake word.
What we learned
- Three gestures that work every time beat five that work most of the time.
- When a device can't talk back, the action itself has to be the feedback.
What's next
- More devices: lights, speakers and TVs through one smart-home adapter.
- Testing with people who have limited hand mobility, who stand to gain the most from one small set of large gestures.
- Recognizing devices without markers or setup.
Built With
- bluetooth
- deepgram
- gemini
- javascript
- mentra
- python
- swift
- vexo