A-Eye

Inspiration

In April 2020 a man named Rob Kenney started a YouTube channel called "Dad, how do I?" His father had left when he was 14, and he filmed the things nobody had shown him: how to tie a tie, how to fix a running toilet, how to check the oil. Over five million people subscribed. Most of the comments say some version of the same thing: I never had anyone to ask.

A video can show you a sink. It cannot see yours. It cannot tell you that the valve under your cabinet is the one, that the water is actually off, or that you should stop because there is a puddle next to the outlet.

So we built the companion you call when there is no one to call. Point your phone at the thing. It looks, it talks, it draws on the picture, and it checks the dangerous step before you take it. Then, on purpose, it teaches you to need it less.

What it does

You open a call and pick what you are doing: fixing a dripping sink, soldering cables, tying a tie. A-eye sees the camera at one frame a second, speaks one step at a time, and draws a box and an arrow on the live picture that stay stuck to the thing as you move.

The app slowly guides you to complete the task that you want to finish and it helps you get unstuck during tough challenges. Through our realtime interaction system, we can ask clarification question and the model will look through our database realtime which has manuals and guides for electronics and procedures that we scraped through APIFY.

Our database (and low latency system)

Every piece of state in A-eye is shared and live: the step you are on, the fade level, the valve flag, the hazard that just revoked it, and the box the tracker is holding on screen. We tried Postgres that meant a table, an API route, a WebSocket relay, and a cache in each client. This would be too slow and complicated for our product.

The valve pass would show as armed on the web page but still be locked on the phone for close to a second. A hazard said out loud could land a step too late. We lost most of an evening writing reconciliation code for a bug that was really an architecture fix.

SpacetimeDB removed the stack instead of patching it. The gate, the hazard stops, the fade counter and the hint tally are reducers that run inside the database, so there is no server process between a client and the rule. A client subscribes to the task row and the next state arrives as a push.

The valve check writes its verdict with one reducer call, the hazard reducer clears it in the same place, and both clients see the same row at the same instant because there is only one row.

The rules we were proudest of, the ones that must never fade and never be overruled by a model, are now the only code path that can touch the flag. That is what we wanted from Postgres and could not get: the safety logic living where the data lives, and nothing in between to drift.

Our users also have governance over their own data and they can choose to delete or download their own data.

What happens under the hood

Each home task is a fixed list of steps with one hint per step, so the model only describes what it sees and never writes instructions. Valve state comes from Qwen 2.5-VL on our GPU as a two-field JSON object, state and confidence, and anything unclear returns "unsure", which never unlocks the next step.

Claude Haiku 5.5 answers first from a caption and object list, Sonnet 5.5 plans the overlay from a single crop, and Opus 5.5 or Fable 5.1 take the rare hard turn. Qwen 3.5 grounds objects about once a second while a browser-side tracker keeps the box on target between frames. Progress is stored as encrypted numbers, or kept in memory when no key is set.

How we built it

We wrote the safety stops first, before any model was wired in. Each hazard became a fixed line in code with a test that fires it from a sentence, and the task files came next: a list of steps and a parallel list of hints, so by the end of hour three the sink could be walked through with no model at all. That ordering was the main decision of the hackathon. Everything a model does was added on top of a product that already refused to unlock step four.

The vision pieces came in the order of their risk. The valve checker went first, Qwen 2.5-VL on our GPU behind a strict two-field schema, and we spent an hour feeding it round handles.

Progress storage was rewritten once. The first version logged sentences; the final version stores two encrypted numbers per run, or keeps them in memory when no key is set, because the only feedback A-eye gives is a sentence built by code from those numbers.

The web and iPhone clients were built in parallel against one set of JSON Schemas, written before either client, so the two teams never waited on each other.

People and coding agents worked under one rulebook for about 24 hours: one owner and one worktree per package, and no merge without a result file naming the test command and whether it passed.

What's next

The next task packs are a car jack, a breaker panel and a kitchen fire, because those are where the stops matter most. Each needs its own checker: a jack-stand check before anyone goes under a car, a breaker-off check before a cover comes off, a flame-size check that says "leave" instead of "smother" past a threshold.

We are also adding a second camera mode for the iPhone that uses LiDAR depth, so the box and arrow can say "the one behind the pipe" and not just "the valve".

The live state already runs in SpacetimeDB, so adding clients is a subscription, not a server. The GPU is the real cost. Grounding at one frame a second per active call is what we have to pay for, so the first scaling work is batching frames across calls on one GPU and dropping to half a frame a second when the tracker is confident.

The first buyers are organizations that already pay a person to teach these skills and cannot put that person in every kitchen: trade schools, transition programs for young people leaving foster care, and independent-living services for older adults. We sell per seat, with task packs built alongside their instructors, and report the one metric they care about, hints per run falling across a cohort. Consumers come second through the iPhone app, once the library is wide enough that the second open is for something new.