Sample of the steps of a task
Supervisor report of the worker
Task accomplishment
Notability file
Notability file
Notability file
Architecture diagram of the workflow
Cartesia Workflow

Inspiration

Many people with intellectual and developmental disabilities (IDD) want to work, and can. But the first weeks at a new job are the hardest. Only 22.7% of people with a disability were employed in 2024, against 65.5% of people without one (US BLS). The support that helps most, a job coach standing next to you, is scarce: the direct support workforce turns over at about 40% a year (National Core Indicators, 2023). A coach might visit twice a week. The other days, small and fixable mistakes add up, and small mistakes cost jobs.

We asked: what if the coach never had to leave? Glasses that teach a task step by step, in the voice of the supervisor the worker already trusts, notice mistakes the moment they happen, and step back as the worker learns, because the goal is independence, not dependence.

What it does

drigil is a job coach in a worker's glasses. Our demo worker is Jordan, at a café counter.

Teach once. The supervisor records a short narrated video of the task. Claude turns it into 4–8 plain steps. Each step has an instruction, what "done right" looks like, the likely mistake, a gentle correction, a reason, praise and a short cue. With the supervisor's consent, their narration becomes a Cartesia voice clone.
Coach live. Jordan presses Start. The glasses camera streams face-blurred frames. Claude checks each one against the whole task and says done, mistake or in progress. Jordan hears the next step, or a fix like "Almost. Turn the sleeve so the words face out.", in his supervisor's voice, with a box drawn around the problem.
Talk. An always-on mic hears "Is this right?", "Why?", "I'm stuck" or "What am I holding?" and answers out loud, using what the camera sees right now.
Remember. Moss keeps short notes about each worker. Next time, before the tricky step: "Remember: turn the sleeve so the words face out."
Fade. Every finished step is recorded. As Jordan masters a step, its coaching shrinks from full instructions to a short cue, to alerts only, to silence. Level-ups are celebrated out loud. Steps back are never announced.
Re-teach (Factory Droid). If one step keeps causing trouble, Droid rewrites that step of the task in a sandbox, for example splitting it in two or writing a clearer correction. The supervisor sees Droid's work live and approves before anything changes.
A person is always one tap away. "Help" pages a coworker buddy first. Two buddies race and exactly one wins. After 20 s it escalates to the supervisor. A hold on the wristband calls the supervisor directly.
The supervisor's dashboard shows who is working now, who needs help, who is getting more independent and which steps need re-teaching, all live.

How we built it

Browser (React + TypeScript): MediaPipe hand tracking and face blurring before any frame leaves the device, an always-on mic with voice detection, and live views for the worker, supervisor and buddy.
Edge app (Python, FastAPI): the coaching brain. It handles the live checks, the dignity rules, voice, memory, and the trigger and runner for Droid.
Anthropic Claude: Opus 5.5 writes the steps from the teach video. Sonnet 5.5 checks every live frame (the whole step list in one call) and answers Jordan's questions using the live image.
Cartesia: sonic-3.6 speech in the cloned supervisor voice, with instructions pre-recorded so playback is instant, ink-whisper speech-to-text, and consented voice cloning from the teach video.
SpacetimeDB: the shared, live workplace. It holds sessions, step outcomes, per-step mastery, help requests with an atomic claim race, wristband calls and Droid's job stream. Reducers enforce the rules: only the edge app can record outcomes and only supervisors can override levels. A scheduled reducer escalates unclaimed help after 20 s. Every screen updates through live subscriptions.
Moss: a per-worker memory, recalled in about 1.4 ms, before each step and when Jordan asks a question.
Factory Droid SDK: the re-teach engineer. It is triggered by data (2 corrections or help requests on one step within an hour) or by the supervisor. It runs at medium autonomy in a per-job sandbox holding only the task, the evidence with names removed, the rules and a validator. Our runner re-validates the output itself, every event streams to the dashboard, and nothing goes live without the supervisor's approval.
Bitdeer AI Model Studio: we used it to benchmark and tune the vision check across catalog models (GLM-5.3-Flash, Qwen3.8-27B) on our labeled test set.
Hardware: a Raspberry Pi 5 with a Logitech Brio 100 is the glasses camera, streamed over USB-C. An Arduino Uno Q with a touch sensor is the wristband (hold to call the supervisor, short touch for a quiet note).

The fading rule lives in one SpacetimeDB transaction. Support level $\ell \in {0,1,2,3}$ goes from full help to on their own:

$$ \ell_{t+1} = \begin{cases} \min(3,\ \ell_t + 1) & \text{if the last 3 runs of this step were clean} \ \max(0,\ \ell_t - 1) & \text{if } \ge 2 \text{ of the last 5 runs had an error} \ \ell_t & \text{otherwise} \end{cases} $$

Dignity is a rule in the code, not a slogan

A wrong correction hurts more than no correction:

Automatic corrections: they need two mistake answers in a row, wait out a 3 s grace period after each new step, and come at most once every 10 s.
Silence when unsure: "unclear" or low-confidence answers say nothing.
Never backward: the coach never sends a worker back to an earlier step.
Plain language: every line is at a first-grade reading level, one action per sentence, and corrections start with "Almost."

Measured, on our 30-frame labeled set (coffee-sleeve task)

Live check	Accuracy	Mistakes caught	False corrections	Median latency
Claude Sonnet 5.5 (live)	73%	44%	6.7%	2.2 s
GLM-5.3-Flash on Bitdeer	77%	56%	3.3%	4.4 s

Other components: Moss recall p50 1.43 ms / p99 2.68 ms · Cartesia first audio 723 ms.
Droid: a real re-teach run took about 35 s with 35 streamed events and passed validation.

Challenges we ran into

Speed vs. accuracy. Bitdeer's GLM-5.3-Flash was our most accurate checker but took about 4.4 s per look, too slow to keep up with a worker. Claude Haiku was fast but missed every "sleeve on backwards" mistake. One model, Claude Sonnet, seeing the whole step list, was the best balance.
Never send a worker backward. Finishing step 5 can hide step 4's result, and the model would ask the worker to go back. We mark earlier steps as done in the prompt and clamp progress in code.
The app hearing itself. With no echo cancellation, the always-on mic heard the coach's own voice. We pause the mic while speaking and drop anything that matches what the app just said.
One voice, always. A stock voice kept slipping in between cloned lines. Every phrase now routes through the consented clone.
Prompts that leaked. Example steps in our teaching prompt showed up in unrelated tasks. Short, generic prompts that use only what is in the video fixed it.
Real-world networking. Phone cameras need HTTPS, and event Wi-Fi blocks device-to-device traffic. We added a local certificate and fell back to a hotspot.
Letting an agent edit the lesson safely. The Droid sandbox, the runner's own validation and the approval step came from asking "what's the worst thing it could do?"

Accomplishments that we're proud of

The whole loop runs end to end on real hardware: teach from a video, coach live in a cloned voice, catch a mistake, answer a question, fade support, page a buddy, call from the wristband, and re-teach with Droid.
Dignity is built into the code (debounce, silence when unsure, steps back never announced), not left to the model.
Privacy by design: faces are blurred in the browser, no video of the worker is stored, and teach videos are deleted after analysis.
Every number we quote is measured, and we label what is simulated.

What we learned

Supported-employment research already has the playbook: task analysis, prompt fading and coworker supports. The best AI design here was to follow it, not reinvent it.
For this community, "when to stay quiet" matters as much as "what to say".
Agents in a real product need boundaries more than intelligence: a narrow sandbox, independent validation and a human approval step.

What's next for drigil

Pilot with a supported-employment program and a few job coaches.
Move the speaker to bone-conduction earbuds on the frame, and route the glasses mic into live talk.
Grow the task library by letting coaches share and remix lessons.
Run the check on-device to cut latency and keep more data local.

Real vs. simulated

Real: the camera checks, cloned voice, speech answers, memory, fading, help routing, wristband, Droid re-teaching and every dashboard number.
Labeled "Sample": the landing page's week-by-week journey.
Seeded examples: the three café practice tasks.
Voice clone: a teammate playing the supervisor, recorded with consent.