Inspiration

Technicians stand inside broken machines with greasy hands and tools in hand—they cannot scroll 180-page PDF manuals or scrub YouTube videos. Standard
voice bots are blind talking heads. We built **Third Hand** to give voice AI spatial awareness: an agent that shares your screen, manipulates a live
3D digital twin mid-sentence, and sees what you are pointing at before it speaks.

## What it does

**Third Hand** is a hands-free, multimodal voice copilot for repairs and engineering labs:
- **The Closed Loop:** Say *"my extruder is clicking"* or *"scooter won't start"*, and the 3D model explodes, focuses, and highlights the faulty part
before the voice finishes speaking.
- **Visual Perception:** Hover any part and ask *"what's this bit?"*—the agent identifies it via 3D pointer tracking (`get_user_pointer`) without you
naming it.
- **Live Web Pricing (Anakin.io):** Quoting real-world replacement costs with interactive source cards and price pills.
- **Multilingual:** Native voice support for 22 Indian languages and Hinglish (Vaani Saaras & Bulbul v3).
- **Environments:** Ender-3 3D Printer, 110 cc Commuter Scooter, and a Pelton Wheel Virtual Lab with an automated oral Viva Voce exam.

## How we built it

- **Vaani Voice AI**
streaming TTS), and LiveKit WebRTC.
- **Anakin.io:** Live web search and page crawling for real-time parts pricing.
- **Frontend:** React 18, Three.js, React Three Fiber, Zustand, and Tailwind CSS with local lighting and zero CDN dependencies.
- **Backend:** FastAPI (Python 3.12) routing tool webhooks and session tokens.

## Challenges we ran into

- **Audio Latency:** Masked speech synthesis delay by streaming scene tools immediately—the 3D model moves in 200 ms while the audio buffers.
- **Bidirectional Perception:** Enabling the agent to query the browser's 3D cursor before answering, rather than just speaking unidirectionally.
- **Nesting 3D Explodes:** Calculating bounding-box depth so interior parts (battery, fuse box) explode cleanly without clipping outer bodywork.

## Accomplishments that we're proud of

- **The "Magic Moment":** Pointing at an unknown part, asking *"what's this?"*, and having the agent identify it and quote live web prices without
hearing its name.
- **Mid-Sentence 3D Sync:** Flawless real-time choreography between spoken advice and 3D camera animations.
- **Offline Reliability:** Procedural 3D assets that render reliably even on poor venue Wi-Fi.

## What we learned

- Voice AI needs spatial grounding—a voice bot that can see and control what the user looks at transforms from a novelty into an essential workshop
tool.
- Fast visual feedback completely eliminates perceived voice latency.

## What's next for Third Hand

- **Camera Vision (Vaani Vision):** Scan physical machine rating plates to auto-load matching 3D models.
- **AR / Smart Glasses:** Projecting exploded diagrams and highlights directly onto the physical hardware.
- **Phone Line Twin:** Calling a toll-free number from any phone using Vaani's telephony platform.