Meanwhile

Tell Alexa what to care about before you step away. Meanwhile watches every camera at once, pings you when it matters, and tells you what actually happened.

Comment

Inspiration

I work from home and people come and go all day: family, a helper, deliveries. Every time I step away I lose the thread. Cameras record everything, but they leave the work to me: scroll motion clips, one camera at a time, and guess what happened.

I wanted the opposite. Say what I care about before I leave, and let the house follow that one thing across every camera, interrupt me only when it matters, and tell me the outcome, not the motion.

What it does

  • Instruct in your own words. "Alexa, I'm stepping away, keep an eye on my keys." Meanwhile finds them and follows them on every camera.
  • Heads-up when it matters. Out of view on all cameras for 4 s → a push on your phone.
  • A verdict, not a clip. When any camera sees it again: "handed back to you, not taken". Picking it up and putting it down in view stays quiet.
  • Knows who. Amazon Rekognition names people: you're "you", and anyone you add from the app with a photo is named too. Anyone else stays "someone".
  • "What did I miss?" One spoken sentence, details if you ask, "show me" plays the clips. "Do you see me?" and "what did I just do?" answer from the cameras now.
  • Any phone is a camera. Open a web page, scan a one-time QR code to pair. Nobody ever sees a key.

How I built it

  • Alexa+ via MCP: a self-hosted MCP server (official Python SDK, spec 2025-11-25, Streamable HTTP) with tools like step_away, what_did_i_miss, show_me, look. Read tools answer in ~23 ms, well under Alexa+'s 500 ms budget. In the demo, a simulated Alexa+ in the Android app drives it end to end (a browser panel does the same on a laptop).
  • Tracking is a detector, not a chatbot. Phones stream over WebRTC; YOLOE (open-vocabulary) + ByteTrack follow the object at 45–55 ms a frame on a T4. Simple geometry rules turn tracks into events.
  • The vision model only when something changes.Qwen3-VL-8B on an Amazon SageMaker endpoint finds the object when the detector loses it, takes one look before a heads-up, and reads clips. On the same recorded run, that cut model calls from 36 to 3 (12 times fewer), with no false events.
  • On AWS: one EC2 GPU box (Caddy HTTPS, core, voice server), SageMaker for vision, Rekognition for faces, plus a Kotlin Android app with Firebase push.

Challenges

  • A vision model can't follow an object. My first version asked a model about every frame. It missed a half-visible controller and credited one person's actions to another. Moving to a detector + tracker fixed it.
  • One word mattered. "controller" boxed my keyboard on 212 frames; "game controller" found it in 909 of ~1030 with no false hits.
  • Small frames lose small things. At 320×180 the tracker never started; an upright phone sent 204×360 and a helmet was never locked. I now seed the track from where the object was at step-away and ask the model when the detector loses it.
  • GPU capacity. No g6 or g5 capacity in Mumbai, so the box runs on a T4; SageMaker quotas showed APPROVED but applied as 0 for five days.
  • Live video crashed the phone app while Chrome played the same stream fine: a WebRTC library release with mismatched Java and native halves.
  • A cloud box lives in UTC. Spoken times were 5½ hours off until the voice side got a home time zone.

What I learned

  • Ask the model the question the rule needs ("is it still at its spot?"), not an open one.
  • A slow answer is worse than none in a live loop.
  • A tracker has to start from what you already know, or a weakly detected object can never go "missing".
  • A real phone is the only honest test of a WebRTC client.
  • Check that every cloud service you plan on is actually available to you on day one.

What's next

Prediction from the camera layout (where it's heading), more standing instructions with their own verdicts (who came in, did I lock the door), a real Alexa+ device on the same MCP server, and a hosted multi-home edition where cost follows active watching.

Built With

Share this project:

Updates

Submission history