Inspiration

A mechanic shouldn't have to stop fixing a car just to find out how to fix it.

CARVIS started with a friend who works as a mechanic. He described a frustrating routine: when he needed a torque specification, he'd put down his tools, wash his hands, walk to the shop computer, open ALLDATA, and search through service documentation. Then he'd return to the vehicle, trying to remember the exact number and the conditions attached to it.

The information was never missing. It was just out of reach.

We started asking a simple question: What if the mechanic never had to leave the vehicle to get the answer?

That question became CARVIS — Conversational Automotive Repair & Visual Intelligence System.

Eyes on the job. Answers in your ear.

What it does

CARVIS transforms Meta Ray-Ban smart glasses into a hands-free automotive copilot that can see what you're working on, understand what you're asking, retrieve verified technical information, and connect the work to your shop.

Imagine working underneath a vehicle and asking:

"Hey CARVIS, what's the tolerance on the ride height of the vehicle?"

CARVIS responds through the glasses, confirms the vehicle, finds the matching specification in its service documentation, and speaks the documented answer. No searching through manuals. No reaching for a phone. No interruption to the job.

We demonstrated this interaction using physical Meta smart glasses.

And that's just one part of the experience. CARVIS combines three core capabilities:

  • SEE — AI-powered visual understanding. Ask CARVIS what you're looking at. NVIDIA Nemotron analyzes images captured through the glasses, identifies likely automotive components, and provides visual context with uncertainty.
  • VERIFY — Answers grounded in real service manuals. Qdrant retrieves vehicle-specific documentation, while our validation engine checks exact applicability, specifications, and sources before delivering technical answers. If a value cannot be verified, CARVIS asks for clarification or refuses to guess.
  • CONNECT — A smarter, connected shop. SpacetimeDB powers shared shop identities, mechanic and manager roles, invitations, and query history, giving teams visibility into the questions asked, answers returned, and sources used.

CARVIS remembers the confirmed vehicle and working context throughout a conversation, allowing mechanics to ask natural follow-up questions without starting over.

The mechanic brings the question. CARVIS brings the context, the evidence, and the answer.

How we built it

We built CARVIS around one principle: AI should adapt to the way people work, not force people to adapt to AI.

The system combines a native iOS experience, wearable hardware, conversational AI, computer vision, a verified knowledge engine, and a real-time shop workspace.

A native, hands-free experience

Our primary interface is a SwiftUI iPhone app connected to Meta Ray-Ban glasses through the Meta Wearables Device Access Toolkit.

The glasses act as the mechanic's eyes, ears, and voice interface. The iOS app manages device connectivity, camera access, vehicle context, conversation state, and visual feedback.

We integrated Vapi for real-time voice conversations, with GPT-4.1-mini coordinating dialogue and tool calls. CARVIS supports wake-phrase activation during an armed session, vehicle confirmation, contextual follow-ups, and short spoken responses designed for someone actively working with their hands.

Instead of building another chat interface, we designed an experience where the most important interaction happens without looking at a screen.

NVIDIA Nemotron — Giving CARVIS eyes

A mechanic can describe a problem, but sometimes showing it is easier.

NVIDIA Nemotron serves as CARVIS's visual intelligence layer, interpreting fresh automotive images and returning component hypotheses, relevant observations, and uncertainty.

The difficult part was connecting a cloud-based AI tool to a camera physically mounted on someone's face.

We built a camera-command pipeline that links Vapi's tool calls to the active iOS session. When CARVIS needs to inspect a component, the backend issues a request to the phone, the glasses capture a fresh image, and the result is routed back to the correct conversation.

We implemented request correlation, timeouts, duplicate protection, and stale-result rejection to coordinate this asynchronous workflow.

Crucially, visual identification and technical verification remain separate. Nemotron helps CARVIS understand what it sees; service documentation determines what specifications can be trusted.

Qdrant — Intelligence grounded in evidence

Automotive specifications leave little room for guesswork. A torque value for the wrong fastener or an oil capacity for the wrong transmission can have real consequences.

We built a structured service-manual knowledge engine using FastAPI, Qdrant, FastEmbed, LlamaParse, and Python.

LlamaParse processes technical documentation into structured records. Each record preserves important context, including vehicle year, make, model, relevant variants, component, operation, exact values and units, source location, and verification status.

When a mechanic asks a technical question, CARVIS:

  1. Identifies the confirmed vehicle and requested operation.
  2. Retrieves matching, reviewed evidence from Qdrant.
  3. Checks applicability, variants, source integrity, and conflicting records.
  4. Returns a precisely formatted, source-backed answer—or asks for clarification when the evidence is insufficient.

This is more than placing a service manual inside a vector database. We designed the retrieval and validation process to distinguish between a result that sounds relevant and a specification that actually applies.

The language model interprets the question. The service manual supplies the facts.

SpacetimeDB — From one mechanic to a connected shop

We wanted CARVIS to be useful beyond a single conversation.

SpacetimeDB powers the collaborative layer of CARVIS, organizing user profiles, shops, membership, manager and mechanic roles, invitations, and shared query results.

We built a reactive shop dashboard where teams can access recorded questions and outcomes, including vehicle context, supporting references, and response latency.

This creates two complementary kinds of intelligence:

Qdrant knows the vehicle. SpacetimeDB connects the people working on it.

Together, they establish the foundation for a shop where useful technical information isn't isolated to one mechanic's interaction.

Challenges we faced

Connecting the physical and digital worlds

Our biggest engineering challenge was making several independent systems behave like one seamless assistant.

The glasses manage camera and audio hardware. The iPhone manages the device session. Vapi manages conversation. Our backend manages context and verification. NVIDIA handles visual inference.

A request might begin as someone's voice, trigger a cloud tool, require a fresh photo from physical hardware, and return as a spoken response—all while the mechanic continues working.

Coordinating those lifecycles required careful management of Bluetooth audio routing, wearable permissions, session state, asynchronous camera requests, and failure recovery.

Making technical AI trustworthy

Retrieving a plausible answer is not the same as retrieving the correct answer.

Oil-only and oil-plus-filter capacities differ. Transmission configurations matter. Nearby components can have completely different specifications. Even two relevant manual records can conflict.

We addressed these problems with structured applicability filters, explicit vehicle confirmation, reviewed source evidence, deterministic numerical formatting, and refusal behavior rather than allowing a language model to improvise technical values.

Making complexity disappear

Behind CARVIS is a multi-service architecture. In front of the mechanic, there should be one simple interaction: ask a question and get a useful answer.

Balancing that technical complexity with a clean, intuitive, voice-first experience was one of our most important design challenges.

Accomplishments we're proud of

  • Brought CARVIS onto physical Meta Ray-Ban glasses, demonstrating hands-free, spoken retrieval of a verified vehicle specification.
  • Integrated NVIDIA Nemotron for automotive visual interpretation and built the camera-command infrastructure connecting wearable capture to cloud AI tools.
  • Created a context-aware conversational system that tracks vehicle details, handles corrections, and supports natural follow-up questions.
  • Built a source-grounded automotive knowledge engine with structured manual ingestion, exact vehicle matching, conflict detection, and unsupported-specification refusal.
  • Developed a SpacetimeDB-powered shop workspace with role-based access, invitations, shared activity records, and reactive views.
  • Validated the implementation with 113 passing backend tests, 66 conversation-state checks, 29 wake-phrase checks, and successful frontend and database-module type checks.

The accomplishment we're proudest of is seeing a question spoken through smart glasses turn into a verified answer spoken back to the mechanic.

What began as a frustrating trip to a shop computer became an entirely different way to interact with technical information.

What we learned

We learned that hands-free is not just a feature. It's a different way to design software.

When someone is working with tools, their attention belongs to the physical task. Good software should respect that attention, remember context, communicate concisely, and appear only when needed.

We also learned that powerful AI systems become more reliable when every component has a clearly defined responsibility.

Nemotron interprets visual information. The conversational model understands intent and coordinates tools. Qdrant retrieves relevant documentation. Python validates technical evidence. SpacetimeDB organizes the people and shared work surrounding those interactions.

The result isn't one model trying to do everything. It's a coordinated system in which each layer contributes what it does best.

What's next for CARVIS

We see CARVIS evolving from a hands-free automotive assistant into an intelligent workspace for technicians and repair shops.

Our next steps include expanding the library of reviewed vehicle manuals, supporting more repair and diagnostic workflows, strengthening visual inspection capabilities, and building richer collaboration features for shop teams.

Longer term, we envision mechanics being able to look at a component, ask questions naturally, follow source-backed procedures, and share useful findings with their team—all through a seamless wearable experience.

The goal isn't to replace a mechanic's expertise. It's to make that expertise more effective by putting the right information within reach, exactly when it's needed.

CARVIS doesn't take the mechanic away from the work. It brings intelligence into the work.

Built With

  • apple-speech
  • avfoundation
  • claude
  • fastapi
  • fastembed
  • llamaparse
  • meta-wearables-device-access-toolkit
  • nvidia-api
  • nvidia-nemotron
  • openai
  • openai-gpt-4.1-mini
  • pydantic
  • pytest
  • python
  • qdrant
  • ray-ban-meta-smart-glasses
  • react
  • spacetimedb
  • swift
  • swiftui
  • typescript
  • uvicorn
  • vapi
  • vite
  • xcode
Share this project:

Updates

Submission history