Inspiration
Experts teach the same task over and over. A recorded tutorial helps, but it plays at its own pace. It cannot see whether you finished the step, so you pause, scrub back and guess. We wanted the recording to wait for the learner.
With HandOver the expert records once, and anyone can learn from it after that.
What it does
The expert records once
Tasks can be recorded on Meta glasses or any camera. HandOver cuts the recording into steps. Each step gets a clip of the expert's hands, captions, a reference frame of the finished state, and a spoken line.
The camera follows along
The learner streams what they are doing from a phone or smart glasses, and the expert's step plays beside it. A vision model watches their progress and moves them to the next step the moment it sees this one done. Next always works too.
Behind? It says the common mistake
If the learner is behind, HandOver compares the live view with the previous, current and next step. If they still match the previous step after a few seconds, it shows and speaks that step's common mistake.
The expert sees where people get stuck
Every check, advance and caught mistake can go to Databricks. The expert sees which steps slow people down and fixes the lesson there. Learners see exactly where they are weakest.
How we built it
Cutting a take into steps
The expert's video goes through one command. A model watches the whole take and marks where each step ends. If the expert narrates, it reads the transcript alongside the frames. If they work quietly, it looks for the first moment each step's finished state appears on screen, then looks again, frame by frame, around that moment to get the cut right. Every step comes out with a clip, captions, a reference frame of the finished state, a yes or no question the camera can answer, and the mistake people usually make there. Before the lesson ships, each question is tested against its own reference frame and the previous step's. If it cannot tell the two apart, the step becomes manual. A narration is written for each step and spoken by a local voice model under the expert's audio.
Fixing a bad cut
The pipeline prints a contact sheet, one row per step. If a cut is wrong, you move the time and recut. The lesson rebuilds from the corrected times without running the model again. The bike lesson was cut by the model for about 3 cents.
The hub
One laptop on the local network runs the lesson. Cameras stream frames to it, and it keeps only the newest frame per camera in memory. It never writes a learner's frame to disk, and a test makes sure of that. A motion gate watches the stream and only sends a frame to the vision model when the hands have gone still and something has changed since the last check, so the model is asked a few times a second at most, not on every frame.
The check
The vision model gets three images: the previous step's reference, this step's reference and the live frame. It has to write what still differs before it answers yes, no or unsure. A step passes on two confident yeses on two different frames, or one very confident yes that names no difference. One yes that still sees a difference never passes a step on its own.
The clients
One hub, five clients. A web app runs the lesson on the laptop. A phone camera page turns any phone into a learner camera from a QR code, with nothing to install. A SwiftUI app for iPhone and iPad, a Tauri desktop app for Mac and an Android app all speak the same contract, so the lesson can be driven from whichever screen is nearest and the camera can be whichever device is pointed at the work.
Meta glasses
The iPhone app bridges Meta's Wearables Device Access Toolkit, so Ray-Ban Meta glasses can sit at both ends of HandOver. The expert records the take first-person, hands free, which is how the bike lesson was made. The learner wears them as the camera, and the step's voice line plays in the glasses' speakers, so their hands never leave the work. We registered a Ray-Ban Meta Gen 2 in a signed development build and streamed its live camera into a running lesson.
Databricks
Every check, advance and caught mistake becomes an event. Events go to a local log first and upload to Databricks in the background. On top of them sit a dashboard, where learners spend their time compared with the expert and how the camera checks are doing per step, and a Genie space that answers questions like "Which step do people get stuck on?" in plain English.
Challenges we ran into
The model said yes too early
A fold that was three-quarters done looked finished to a single check. We measured it on drawings of folding stages, and one check passed 10 of 63 unfinished frames. So the model now sees the previous step as well as the current one, and a step needs two agreeing yeses on two different frames before it passes.
Experts who do not narrate
Our first cutter found steps from what the expert said. On the bike take there are long quiet stretches with nothing to transcribe, so we built a cutter that watches for the moment each step's finished state first appears.
Some steps are invisible to a camera
Feeling a rim for wear or squeezing a brake lever does not change the picture. In rehearsals the model answered unsure on the bike steps rather than guessing. So a step can be marked manual, with no camera check at all, and the learner presses Next. The bike lesson ships with two of those.
Keeping analytics out of the live loop
A check that waits on a warehouse is a check the learner waits on. Events go to a local log and upload in the background, and a test slows the warehouse on purpose to confirm the lesson does not notice.
Accomplishments that we're proud of
Latency
Across 19 team rehearsal runs, none of the 185 live checks errored or timed out. The median time is about 1.1 s from action to feedback.
Cost
2,318 model calls for 19 cents. Cutting a three-minute take into a lesson costs about 3 cents.
The edit loop
One command turns a raw take into a narrated, captioned lesson with camera checks and a self-test. When a cut is wrong, you move a time and recut. Nothing has to be re-recorded or re-processed by the model.
What we learned
- Check questions must describe what a camera can see: shapes, seams and counts, never color.
- Marking a step manual beats a check that passes by accident.
- Analytics belong outside the live loop.
What's next for HandOver
- A hosted hub, so a team of learners can share lessons.
- Feedback shown on the smart glasses themselves instead of a laptop or phone screen.
- Accuracy numbers from people new to a task, then a real workplace task taught by a working expert.
Built With
- ai-bi-dashboards
- anthropic
- claude-haiku
- claude-sonnet
- databricks
- databricks-genie
- fastapi
- faster-whisper
- ffmpeg
- google-gemini
- kokoro
- python
- react
- rust
- swift
- swiftui
- tauri
- typescript
- unity-catalog
- websockets