Spanner: submission texts
What Spanner is, and what it actually uses. Each section describes only things the running product does, and names the file where it happens.
Tracks: Voice, Vision, Generative. Bounty: ElevenLabs. Big prize: Figma (Best-Designed AI Product).
Repo: https://github.com/ypghanate/Spanner Figma file: https://www.figma.com/design/9t8Cp1iX4xGTL48QcRHsr8
One-liner (use everywhere)
Spanner is a repair coach that watches through your phone's camera, listens to what you say is wrong, shows the exact step you are on as a 3D hologram, talks you through it, warns you when you make a mistake, and finds a replacement part when you ask for one.
Project story
Point your phone at what's broken, and it talks you through the fix, step by step, hands-free.
Inspiration
Most people can't fix simple broken things. They don't know the part names, and text guides and 20-minute videos don't help when your hands are full and you can't tell which of two identical screws the narrator means. We wanted a repair expert who looks at what you are looking at and tells you, in real time, what to do next. "Jarvis, but for repairs."
We also wanted it to be honest about danger. A dead plug fuse is an easy fix, but only if the plug is out of the wall. Spanner needed to know when to say "stop, don't do that."
What it does
Open the Spanner app on an iPhone, point it at the problem and say what's wrong. Spanner:
- listens to you, and understands "the fuse in my plug has blown" without a menu;
- generates a repair for that exact fault, with Claude, in your words, using reference steps from a Databricks knowledge base;
- shows the current step as a 3D hologram on a laptop beside you, taking the device apart to the part you need;
- watches the camera and only moves on when it can see the device and the step is done;
- speaks to you in a natural voice (ElevenLabs), and warns you when it sees a mistake;
- finds a replacement part when you ask for one (Linkup and Tavily), searching only for what you asked to order.
It has guided repairs for a mechanical keyboard, an iPhone 15, an iPad, Sennheiser Momentum 4 headphones and a modern moulded UK 13 A plug (the fuse-swap demo).
How we built it
- Phone: Expo and React Native. It streams camera frames (rear or selfie camera) and cuts speech into short clips using an adaptive noise floor. Tapping the screen cuts Spanner off and makes the next thing you say count as meant for it.
- Backend: FastAPI with Pydantic contracts (
backend/coach.py). - Claude: Haiku 5.5 for conversation and intent, the per-frame step check and shop requests; Sonnet 5.5 for repair planning and the zoomed second look at a frame. Plans are written as structured output, then checked against the real parts of the 3D model.
- Voice: ElevenLabs for speech out and for transcribing what you say (
backend/voice/). - Eyes: frames are resized and sent to Claude. For the keyboard, YOLO finds it and crops to it so key legends are readable (
backend/vision/detect.py). - Laptop viewer: Three.js, with every hologram built procedurally from named parts so a plan can say "highlight
fuse_door" (training/viewer/). - Knowledge: a Databricks Vector Search index of repair steps that the planner searches for reference steps (
training/kb.py). - Parts: Linkup and Tavily for live part search (
backend/shop.py,backend/parts_agent.py). - Safety: the planner refuses scorched, sparking, hot or damaged plugs and always makes step one "unplug it."
Every stage can also run on fixtures with no API keys (USE_MOCKS=1), so each teammate could build in isolation and merge later.
Challenges
Latency. A turn is roughly
$$T_{\text{turn}} = T_{\text{stt}} + T_{\text{reply}} + T_{\text{tts}},$$
and we measured $T_{\text{stt}} \approx 0.4\text{–}1.0\ \text{s}$ and $T_{\text{reply}} \approx 1.1\text{–}1.8\ \text{s}$ before the voice even starts. We cut clips shorter, moved intent checks to Haiku, cached common plans per fault and sent small photos. Phone uploads had been multi-megabyte JPEGs, which hung and failed until we shrank them.
Deciding when a step is really done. One lucky camera frame made the guide jump forward. Requiring two agreeing frames helps. If a wrong read were independent with probability $p$ per frame, two in a row would cut it to about $p^2$. We also made the repair wait until the device was actually in view.
It heard itself, and it ignored us. Two opposite bugs. We then measured how it discards speech. Claude itself dropped 1 of 26 realistic phrases, but our laptop echo filter dropped about 30% of what a person really says. Natural phrases like "lift the old fuse out of the clips" repeat the step's wording, so they matched Spanner's own speech. An echo test based on word overlap,
$$\frac{|W_{\text{heard}} \cap W_{\text{spoken}}|}{|W_{\text{heard}}|} \ge \theta,$$
is only safe when $\theta$ is high and the phrase is long enough, and ours was neither. We made it match only Spanner's own words in order, for five words or more, while it is speaking or just after. On the same test it now drops 0 of 25 real phrases and still catches 6 of 7 genuine echoes, and tapping the screen overrides it entirely.
Merging four people's work. After one merge, the camera handler called helper functions that no longer existed, so every frame crashed once a repair was chosen. A prompt change that said "assume it's for you" quietly stopped plans being generated.
Getting the model right. We first built a rewireable plug with a cover screw and cord grip, then realised modern plugs are sealed, with the fuse behind a small snap-shut door. We rebuilt the model and the repair around that.
What we learned
- Measure, don't guess. Running 24 realistic phrases through the filter showed the real cause in minutes.
- Prompts are code. A one-paragraph change altered what the product does, so prompts need tests like any other file.
- Gate actions on evidence. Waiting for the camera, two agreeing frames and a refusal list make the guide trustworthy.
- Static checks catch merges. A simple undefined-name check would have caught the broken camera handler before anyone ran it.
- Voice products live in the details. Mute timing, echo and when to stay quiet matter as much as the model.
What's next
Move echo handling to the server, add an intent window so it knows when you mean it, and add more repairs beyond phones, headphones and plugs.
Big prize: Figma (Best-Designed AI Product)
The design is one system across the phone, the laptop and the deck, and it is built from the product, not drawn separately from it.
- The phone app follows the Figma file. Its colours, spacing and radii are read from the file's
spanner.*variables (app/src/theme/index.tsx), in light and dark. The 2D step guide artwork is exported from the Figma file'sDiagram_2D_LargeandDiagram_2D_Compactframes (app/src/components/diagram.tsx,diagramAssets.ts). - A design kit built from the running product (
figma/).tokens.jsonholds the laptop viewer's colours, type, spacing, radii and glows, exported straight from its CSS so Figma variables and code never drift apart.renders/holds 14 captures of the product in every state (assembled, exploded, x-ray, each repair step, a mistake, the part inspector, shopping, specs).figma/README.mdis the import guide: tokens into Figma variables, states into a storyboard, components with variants for the step card, camera panel, shop panel and tags. - One system, three surfaces. The kit is set up so the phone, the hologram viewer and the pitch deck share one set of tokens.
- Voice-first design. There is no menu to learn: the laptop asks "What needs fixing?", you answer out loud, and one primary action stays on screen. The call screen shows what Spanner heard, what it is saying, the current step and a single STOP when something is wrong.
- Design for a repair, not a demo. The hologram highlights the exact part, the HUD only appears while guiding, the camera is the background of the call, and tapping anywhere cuts Spanner off and listens. Mistakes are shown in red and spoken at once.
- A storyboard that doubles as a backup demo. The 14 renders are ordered to play a whole repair frame by frame if the live run fails.
Track: Voice, Vision and Generative Models
The whole product is operated by talking.
- In. The phone listens all the time with a noise gate that adapts to the room, cuts each utterance into a short clip, and the clip is transcribed by ElevenLabs (
app/src/lib/voice.ts,backend/voice/stt.py,/coach/utterance). - Understood. Claude Haiku turns the words into one action: plan a repair, start, next, back, repeat, take it apart, show a part, shop for a part, stop, or just talk (
backend/coach.py, prompt inbackend/brain/prompts/converse.md). "Next" and "stop" skip Claude and act at once. - Out. Every reply is spoken in an ElevenLabs voice, and a repair ends with a closing line and the call hangs up by itself.
- Conversation, not commands. It keeps the recent conversation, trusts what you tell it ("there's no screw") over the written plan, and asks one short question only when it has to.
- Talking over it. Tap the screen and Spanner stops and listens; your next words are always treated as meant for it. The phone mutes its mic the moment a reply exists and for a moment after it ends, so Spanner does not hear itself.
Spanner checks what it sees before it moves on.
- Looks. The phone camera (rear or selfie) sends a frame about every second, as soon as the last answer lands. Claude Haiku reads each frame with the repair plan in context and returns: is the device visible, which step it shows, whether the step is done, and whether a known mistake is happening (
_ask,_focus,/frameinbackend/coach.py). - Decides. The guide does not advance unless the device is in view, and a "done" or a "mistake" needs two agreeing frames. This removes the lucky-frame jumps.
- Zooms in when it matters. For the keyboard, YOLO finds it and crops to it so key legends are readable (
backend/vision/detect.py), and a Sonnet second look reads a missing key. - Answers about the picture. Ask "can you see it?" and the latest camera frame goes to Claude with your question.
- Checks the safety. A mistake is flagged out loud, and the planner refuses unsafe repairs before any step is shown.
Nothing about the repair is a fixed script: it is generated from what you said.
- The repair itself. Claude Sonnet writes the steps for the exact fault you describe, each with an instruction, what the camera should see when it is done, common mistakes and which parts it works on (
backend/planner.py,backend/device_planner.py). It is grounded in reference steps from real repair guides and checked against the model's real part names. - The hologram's moves. Each step carries generated 3D commands (take apart, highlight, pulse, reset) assembled from the plan and the device's parts, so a different fault plays a different animation.
- The 3D models. Every hologram is generated procedurally in Three.js from real dimensions, with one named node per part: a 255-part keyboard, iPhone 15, iPad, Momentum 4 headphones and the moulded plug (
training/viewer/viewer.js,device.js). - The voice. Claude writes the short spoken replies, and ElevenLabs speaks them.
- The parts cards. What to buy, and for which device on screen, is worked out from what you asked and then searched live.
Bounty: ElevenLabs
ElevenLabs is both ears and voice, and the voice is the product.
- Speech to text. Every phrase said into the phone is transcribed by ElevenLabs (
backend/voice/stt.py). Measured: about 0.4 to 1.0 seconds per clip. - Text to speech. All of Spanner's speech is an ElevenLabs voice (
backend/voice/tts.py,/coach/say). We made it feel quick: the line is generated and cached the moment a reply exists, so the laptop plays it from disk, and the speaking speed is set for a natural pace. - One voice only. If the voice is slow, Spanner retries once and then stays silent rather than falling back to a robotic browser voice.
- Voice is the whole interface. You start a repair, step through it, take the hologram apart, ask for a part and end the call without touching the screen.
- Built around speech. Echo handling, tap-to-talk and early mute exist because the phone microphone sits next to the laptop speaker.
Also used
- Anthropic Claude (Haiku 5.5, Sonnet 5.5): conversation, planning, frame checks.
- Databricks: Vector Search of repair steps used by the planner, and mistakes and live latency written to Databricks tables, with a dashboard and Genie over them (
training/kb.py,training/feedback.py,databricks/). - Linkup and Tavily: live part search and price and photo recovery (
backend/shop.py,backend/parts_agent.py). - Expo, Three.js, FastAPI, YOLO: the phone app, the hologram, the server and the keyboard detector.
What we did not use. Fine-tuning with Adaption (experiments are in training/, but the live step check runs on Claude Haiku) and fal image generation (switched off).
Pitch
"You can't fix what you can't name, and a video can't see your hands. Spanner can. Point your phone at what's broken and tell it what's wrong, in your own words. It generates the repair for that exact fault, shows you the part on a 3D hologram, and talks you through it in an ElevenLabs voice. Its camera watches your hands, and it only moves on when it can see the step is done. If you get it wrong, it tells you before it costs you. Voice, vision and generation in one loop, with one design system from the phone to the hologram to this deck."
Built With
- chromadb
- claude
- databricks
- elevenlabs
- fal
- fastapi
- faster-whisper
- figma
- gemini
- javascript
- linkup
- mlflow
- numpy
- opencv
- python
- tavily
- websocket
- yolo