Shop Relay

Voice-assisted shop updates that become reviewed pickup plans, customer messages, and copy-ready shop notes.

Comment

In short. A mechanic under a car reports "Maria's Camry. The alternator won't arrive until Friday. Move the pickup and let Maria know." Shop Relay shows what it heard and understood (checkpoint 1), picks a new pickup window from the parts ETA, the remaining labour and the shop's recorded pickup capacity, and drafts the calendar change, the customer email and a shop-system note. Nothing happens until the owner approves that exact plan (checkpoint 2). When two delayed jobs want the same Friday slot, the first approval takes it; the older plan is refused with nothing changed and nothing sent, and when the owner reports that delay again, Shop Relay proposes Saturday. Whatever is still owed, such as a note to type into the shop system, is read back from the records after a reload or restart. In the demo every job is synthetic and the calendar and email are simulated.

Inspiration

In a small repair shop, the person fixing the car is often also the scheduler, the parts chaser and the one who calls customers. When a part is late they stop work, find the job, check who else is picking up that afternoon, tell the customer and update the shop system — usually with gloves on, under a car, in a loud shop. Each step is small; together they pull the owner off the car several times a day, and a missed message means a customer arriving for a car that isn't ready. Shop Relay is built around one real reference shop (an owner and two mechanics); every job and person in the project is synthetic.

What it does

Shop Relay turns a short spoken or typed job update into a plan the owner reviews and approves:

  1. Say it or type it. Tap Push to talk and say "Maria's Camry. The alternator won't arrive until Friday. Move the pickup and let Maria know." Speech is transcribed on the shop's own computer (local Whisper model); audio is not uploaded or saved.
  2. Checkpoint 1 — did it understand me? The page shows I heard (the words exactly as recognized) and Shop Relay understands (customer, vehicle, job, part, arrival day, anything it is unsure about). Edit, Try again or Continue; nothing is submitted before Continue.
  3. It asks instead of guessing. Two Whitfields drive Hondas, so "the Whitfield rotors" gets a question. A name or vehicle no job has, a contradicting vehicle, two different days, or a day with no recorded pickup windows also ends in a question, not a plan.
  4. A proposed plan, in full. From the parts ETA plus remaining labour and the shop's recorded pickup windows, Shop Relay proposes a new pickup, the calendar change, the exact customer email and a copy-ready shop-system note, with the reasons (for example "Fri 4–5 PM: full").
  5. Checkpoint 2 — do I approve this exact plan? Approval is bound to the proposal's revision and content digest. Voice never approves; only the Approve exactly this tap does.
  6. Carried out durably. A Temporal workflow updates the shop record, the calendar and sends one email; PostgreSQL keeps every action and receipt. Approving the same plan twice starts nothing.
  7. Stale plans are refused. If another approval took the proposed pickup window first, approving the older plan changes nothing and says why: "Dan's pickup was not changed, and no email was sent. Fri 10/9 5:00 PM–5:30 PM PDT went to Maria Delgado's pickup after this proposal was made." The owner reports the delay again and reviews a fresh plan (Saturday). Shop Relay does not replan on its own.
  8. Nothing gets lost. The shop-system note is copy-ready for manual entry and stays on the owner's list until they confirm they typed it in. After a reload, a restart or on another device, a new session lists what still needs the owner, read from the records.

How we built it

  • Self-hosted MCP server (Python SDK mcp==1.30.0) speaking protocol 2025-11-25 over Streamable HTTP, with Origin and protocol-header checks and 17 narrow tools in three classes: read, propose and execute. The execute tools are the only path to an external effect.
  • Simulated Alexa+ experience: the owner page is an Alexa-style simulator that is itself a real MCP client of that server (initialize → tools/list → tools/call over Streamable HTTP), labelled "not an official Alexa integration". A details panel shows the negotiated protocol, digests, receipts and the wire trace.
  • Voice layer: browser MediaRecorder → the local server → faster-whisper (base, CPU int8) with the open jobs' names, vehicles and parts as spelling hints → a deterministic understanding preview → owner confirmation → the same report_parts_delay tool as typing.
  • Deterministic core: interpretation only produces a strict intent; plain code matches jobs (with an ambiguity margin and a vehicle-contradiction block), computes readiness and availability (an unrecorded day is unknown, never free), chooses the window, builds the message and routes email only to an allowlisted test inbox. Every proposal carries its basis (job, commitment, availability and provider-environment versions), re-checked at approval and at execution.
  • Temporal + PostgreSQL: one workflow per approval (exec-<approval_id>, duplicate starts rejected); provider calls only inside activities; a reconciler restarts approvals that never started. Calendar writes use a client-assigned event id plus an action marker; an email whose acknowledgment is lost becomes "unknown" and waits for the owner instead of being resent.
  • Provider adapters, each selected independently and labelled on the page: simulated calendar, mail and a deterministic keyword parser (the default, used by the reviewer path); Google Calendar (calendar.app.created, a dedicated synthetic calendar) and Gmail (gmail.send, one allowlisted recipient) through one Desktop OAuth consent; Amazon Bedrock (Converse, Claude Haiku 4.5 through the US inference profile) behind an exact-count spend guard (no-charge CountTokens, atomic reservation against a dollar cap).
  • Reviewer path: scripts\demo.cmd story builds a fresh environment from pinned dependencies, forces every provider to mock and starts processes it owns; no account or credential needed.
  • How the work was done: Dee directed the project; Claude Code (Anthropic) wrote the code, tests and documents under Dee's per-step authorizations; fresh OpenAI Codex processes reviewed it independently and re-verified each fix.

Challenges we ran into

  • Voice in a loud shop. A speech model cannot recover audio that was never captured: a laptop microphone a few feet away produced fluent nonsense on synthetic shop noise, while a close microphone with push-to-talk worked. Even then the browser once recorded from the laptop microphone while a wireless one was selected. So capture comes first (push-to-talk, close microphone, level meter, retry) and a human checkpoint follows transcription.
  • Correcting without guessing. Rewriting a rough transcript into fluent English can make a wrong reading look confident. Independent reviews found phrasings that resolved the wrong job or the wrong time ("Alice, the Camry's alternator…", "Thursday at 3", "roughly four"); each now ends in a question, and each fix was re-verified by a fresh reviewer.
  • Exactly-once is not available from real providers. Idempotent calendar writes reconciled by content and marker, at-most-once email with owner review, and a note that is never "done" until a person confirms it.
  • No live model inference yet. The one authorized Bedrock call was throttled before inference because every Anthropic inference quota on the new account was 0 ($0.00 spent). That was the first blocker reached, not proof that the rest works: the model's real tool call, schema validation and grounding against a live answer are still unobserved, so the reviewer path uses the deterministic parser.
  • A dependency broke silently. A new PyAV release removed an argument the speech library still passes; unit tests with a fake model passed while every real recording failed. A headless-browser check with a fake microphone caught it.

Accomplishments that we're proud of

  • The whole owner loop works end to end on real infrastructure: a real MCP server and client, real PostgreSQL and Temporal, and real local speech recognition, with only the calendar, mail and interpretation simulated in the reviewer path.
  • A three-customer story where one plan goes stale is refused with no side effect and a fresh plan follows — demonstrated in the page and covered by an automated test against the real services.
  • One owner-authorized live run against Google (2026-10-13): one event on a dedicated test calendar and one Gmail send to the allowlisted test inbox, with no duplicate after a reload and a restart (an observed result, not a delivery guarantee).
  • 422 automated tests, strict typing, and repeated rounds of independent review and re-verification.

What we learned

  • Two checkpoints beat one clever model: "did it understand me?" and "do I approve this plan?" are different questions, and a mechanic answers each in seconds.
  • Put speech recognition and language models at the edge; let deterministic code decide everything that touches a customer.
  • Bind approval to content, not to a button: revision + digest + basis makes "approve exactly this" checkable after a restart, and makes a stale plan detectable.
  • "Unknown" is a real state; showing it plainly is better than a retry that might email a customer twice.

What's next for Shop Relay

  1. A supervised field test in the reference shop with real voices and the owner's own pickup rules and wording.
  2. A Bedrock-interpreted run once the account's inference quotas allow it.
  3. A small pilot on real jobs where every message is approved by the owner; text messages (SMS) alongside email.
  4. An Alexa+ connection when the official tools are available to developers; the MCP server already speaks the required protocol.

Limits, stated plainly

  • The demo uses synthetic jobs; calendar and email are simulated in the reviewer path, and interpretation is a deterministic keyword parser, not a language model.
  • Speech recognition was measured on synthetic speech with synthetic shop noise (docs/evidence/voice_eval_2026-10-12.md); no accuracy figure for real voices in a real shop is claimed.
  • The Bedrock adapter is implemented and tested offline but has not completed a live inference.
  • Shop-system notes are entered by hand; there is no shop-system integration.
  • Not an official Alexa integration, not production software, no SMS, no unattended replanning, and no exactly-once guarantee for real external delivery.

Built With

Share this project:

Updates

Submission history