Second Hands

Inspiration

A friend of mine runs a phone repair shop . The most expensive mistakes there aren't hard ones. They're small ones: an apprentice unplugs the display while the battery is still connected and shorts the board, or drives a long screw into a short hole and cracks the logic board. The owner can't watch every bench all day, and a YouTube video can't see what your hands are doing.

The same thing happens everywhere people work with their hands: flat fingers in front of a knife, a bookshelf without its anti-tip strap. So I wanted an expert that sits above the bench, watches every step, and only speaks up when it matters.

What it does

Second Hands is an AI that watches your hands through a phone camera and says "Stop" before a mistake costs you a device or a finger.

1. Start. Type the job ("Redmi Note 8 battery replacement", "Dice an onion"). Tavily finds real guides on the web, and Nemotron Super 120B turns them into steps a camera can check from above: what "done" looks like, the costly mistakes to watch for, and where to put the camera. Dangerous jobs (mains electrical, gas, brakes, firearms) are refused.
2. Pair. Scan the QR with any phone, Android or iPhone. It's an installable web app, so there's no app store and no login.
3. Watch. The phone sends a frame when the scene changes. Nemotron Super VL reads each frame against the current step: step state, a possible mistake, evidence, and a focus box around your hands.
4. Catch. A mistake has to appear in 2 of the last 3 frames. Nemotron Nano decides whether to interrupt. On high-severity steps, Nemotron Ultra reasons over the evidence and confirms it before the phone chimes: "Stop. Disconnect the battery connector first." When it's unsure, it asks instead ("Is that the long screw?"), and you can always answer "I'm right".
5. Report. Steps done, mistakes caught, false alarms, cost, and the frame behind every warning.

Your hands are busy, so it's hands-free: voice commands ("next", "back", "check", "pause") plus big Back / Check / Skip buttons. A laptop dashboard shows the live view, the checklist, the evidence next to the manual, and every model call with its latency and cost.

How I built it

• Next.js 15 + TypeScript + Tailwind on Vercel, Upstash Redis for sessions, reports, rate limits and the daily budget.
• Every model call goes through Nebius Token Factory at runtime. Four NVIDIA Nemotron models each get one job, sized to it:

• Reliability is code, not hope: the 2-of-3 confirmation is deterministic and unit-tested, the vision model can only flag mistake types that the guide lists, every response is schema-checked with one retry, and repeated warnings are suppressed.
• Thinking only where it pays: Eyes, Gatekeeper and Planner run with reasoning off for speed; only the Judge thinks.
• Measured on Token Factory (smoke test, Oct 26): Super VL reads a 768 px frame in 0.8–1.4 s for about $0.0005; Nano gates in ~0.4 s; Ultra verifies in ~1.1 s. A staged "display flex unplugged while the battery is connected" became an Ultra-verified STOP on the second frame.
• Cost stays tiny by design: frames are sent only on change and never faster than one per 1.6 s, so

$$\text{cost per 10-minute session} \le 10 \times 37.5 \times 0.0005 \approx 0.19 \text{ USD}$$

• The phone side uses the browser's camera, speech synthesis and speech recognition, a QR reader (native, with a jsQR fallback for iPhone), the Wake Lock API and a service worker. The dashboard gets a separate ~4 KB preview every 0.7 s, so it stays about 1 s behind the camera without sending extra frames to a model.

Challenges I ran into

• The vision model went silent. On the first real run, Super VL returned empty answers. Nemotron reasons by default, and with a small token budget it spent everything on thinking and returned content: null. Turning reasoning off per call fixed it and halved latency.
• Honest models break fake demos. My zero-setup /try page used a drawn placeholder clip. The real model correctly said "this is a diagram, no hands are visible" and never advanced. Until the real sample clip is recorded, /try plays a clearly labelled recorded run instead.
• "Before" vs. latency. A catch needs two frames plus verification, about 3–5 s end to end. The 2-of-3 rule costs a little speed, but a false "Stop" destroys trust faster than a slightly late one.
• Keeping a public demo alive until judging ends. Per-IP rate limits, a daily spend cap, a 5-minute cap per live session, input size limits (so an oversized guide can't drain the budget), prompt-injection guards on web content, and security headers.
• A lagging dashboard. It first showed only the frames sent to the model (every 1.6–6 s). A separate small preview stream fixed it without touching the model budget.

Accomplishments that i'm proud of

• A full real-time loop (camera → vision → confirmation → gate → verification → voice) running on four Nemotron models in about a second per frame.
• It works on Android, iPhone and PC in any pairing, with no app store and no login.
• Every number I publish is measured, and every mock or replay is labelled as one.

What i learned

• Match the model to the job. A 550B judge on every frame would be slow and expensive; on the rare high-stakes call, it's worth it.
• Determinism around the model (confirmation windows, allowed mistake types, schemas) does more for reliability than prompt tweaks.
• Measure on the real API early. Defaults like reasoning-on change everything.

What's next for Second Hands

• A measured eval on recorded real repairs: catch rate, false alarms, latency.
• Hindi and multilingual voice for repair shops in India.
• Phone-only mode, highlight clips of every catch, and shop mode (one owner, many benches).
• On-device vision on Jetson-class edge hardware for sub-second, offline warnings.

Built With

• getusermedia
• jsqr
• nebius-token-factory
• next.js
• node.js
• nvidia-nemotron
• pwa
• react
• redis
• service-worker
• shadcn-ui
• tailwindcss
• tavily
• typescript
• upstash-redis
• vercel
• vitest
• wake-lock-api
• web-speech-api
• zod