Grand Fix Auto

Rehearse a car repair in VR on a virtual copy of your car, then do it for real in AR. Verified steps, safety order enforced in code, and an AI coach beside you.

Comment

Inspiration

A dead 12 V battery is one of the most common reasons a car won't start, and swapping it is a job most owners could do themselves. Most don't. The steps are scattered across manuals and videos, the order matters (take the negative cable off first, or the wrench can short against the body), and a mistake on a real car is expensive. So people pay shop labor for a 20-minute job. We wanted a way to practice the real procedure safely first, then be guided through it on the actual car.

What it does

Grand Fix Auto is a WebXR app for the Meta Quest 3S. Press Start in headset and everything happens inside VR:

  1. Choose your car and a job. The in-VR menu lists cars and jobs and ai search in backend works for new car and job, and every job is rated for difficulty and risk.
  2. See what you need. A parts-and-tools list for that job and car. Shop on the web page takes you out of VR to the same list on the page, with a shopping search for each item. You can also plan the whole job on the web page in four steps.
  3. Practice in VR. Do the job with your hands or controllers on a realistic car in a home garage: loosen the negative terminal first, lift the old battery out, torque the hold-down. Five practice jobs: 12 V battery, key fob battery, engine air filter, oil check and top-up, and front brake pads.
    • Safety is enforced in code, not by the AI. Step order and safety interlocks come from the procedure. A wrong move, such as the positive terminal first, stops you with a red warning and a spoken explanation.
    • AI coach (online build): Ask coach on a step explains why it matters and what can go wrong. For the danger-rated brake job, code declines and recommends a professional.
  4. Do it for real in AR (12 V battery job, on the verified RAV4). Hands-on mode shows the steps over your real engine bay through passthrough, with markers you place on the battery and terminals. In the online build, an AI camera check looks at a headset camera frame and judges whether the current step is done; you can always confirm a step yourself with Mark done. The other four jobs are VR practice only, and the brake job never leaves VR.
  5. Works offline. The whole app, with about 1,060 pre-recorded voice clips for its fixed lines, is cached on the headset. Practice runs in a garage with no Wi-Fi and no laptop.

How we built it

  • App: three.js r187 + Vite, WebXR in the Quest Browser. Practice runs in an immersive-ar session behind an opaque virtual garage, with a Room view passthrough toggle. Controllers and tracked hands both work, with pinch debounce, a one-euro filter on the hand rays, and poke-to-press.
  • Procedures: research agents (Claude) compiled each job from owner's-manual pages, Toyota technical instructions and recall documents, plus parts-maker and DIY guides where Toyota's documents are silent. A second pass fact-checked every step, and each step carries its source citations and a confidence level. The 12 V battery job for the 2013–2018 gasoline Toyota RAV4 has 41 source-cited steps (29 rated high confidence; the rest are marked medium, low or conflicting), with a 14-step guided path for the demo. The job catalog rates 38 jobs.
  • Step engine: code owns step order and safety interlocks. Claude explains steps and judges camera frames, and its verdict can only move you to the next step in the fixed order.
  • Voice: a Microsoft neural voice, pre-rendered for every fixed line and served by a service worker when offline.
  • 3D: the engine bay, battery area and every interactive part are generated in code from the RAV4's published dimensions, and the same bay sits under every car in the menu. A CC BY car model (with interior and glass) is fitted over it, inside a CC BY home-garage model, with a code-built street outside the garage door.
  • Testing: 670 automated tests (vitest), the Meta IWER emulator on desktop, and runs on the real Quest 3S throughout the weekend.

Challenges we ran into

  • Passthrough camera access in the Quest Browser, a mostly undocumented path. We confirmed getUserMedia with the headset-cameras permission works on the Quest 3S.
  • Hand-tracking reliability: pinch flicker and short tracking losses broke grabs, so we added debounce, grace periods and filtering.
  • Fitting a downloaded car model over our code-generated engine bay without moving the safety-critical positions.
  • Frame rate: keeping 86–90 fps on a Quest 3S with a full garage, a detailed car with interior, and a working engine bay.
  • No internet on the headset: the controller and hand models, the app and the voice all had to work offline.

Accomplishments that we're proud of

  • Safety logic lives in code. A source-cited, fact-checked procedure drives it.
  • Runs fully offline on the headset, with a natural voice.
  • Tested hard: 670 automated tests, plus repeated runs on the real headset at 86–90 fps.

What we learned

  • WebXR on Quest is a fast loop for iterating on real hardware: save a file, reload in the headset.
  • In XR, small details like debounce, hit padding and tracking grace decide whether hands feel natural.
  • Putting an LLM next to safety-critical steps works best when the LLM explains and checks, and code decides the order.

What's next for Grand Fix Auto

  • Verified procedures for more cars beyond the RAV4, and AI-generated 3D models for any car.
  • Hands-on AR for more jobs, and a smart-glasses version of hands-on mode.
  • More jobs, and a parts-to-cart integration.

Notes for judges

  • AI is off in this build. The Claude API key is removed from the submission build, so Ask coach and the AI camera check say "AI API removed for the submission demo". With an Anthropic key on the laptop's local proxy they call Claude. That path is covered by unit tests with a stubbed network, but we did not run it live with a key this weekend.
  • Only the RAV4 is verified. Procedures and specs are verified for the 2013–2018 gasoline Toyota RAV4. The default car, a 2014 Toyota Corolla (the car our 3D model shows), and the other cars in the menu are demo cars that reuse the RAV4 steps for practice. The app labels them that way, and they never get hands-on mode.
  • Adding a car outside the list (AI research plus a generated 3D model) is future work; the page only mentions it.
  • The trailer shows the AI features as designed; they are off in the judged build. "AI learns the repair" refers to the Claude research agents that compiled and fact-checked the procedures during the hackathon. The trailer's AR shot is stock footage with our AR overlay composited on top.

Credits

  • "2014 Toyota Corolla E180 EU (with interior)" by Armored Wave (Sketchfab), CC BY 4.0. Textures compressed, recolored, hood area cut out.
  • "Garage Render Redone FREE" by Ethica (Sketchfab), CC BY 4.0. Scaled, textures compressed.
  • Libraries: three.js, troika-three-text, Meta IWER, @webxr-input-profiles controller and hand models (MIT); Anthropic TypeScript SDK (laptop proxy only). Font: Inter (OFL).
  • App voice: Microsoft en-US-AvaNeural, pre-rendered with the unofficial edge-tts client (clips not in the repo); macOS say as fallback.
  • Trailer AR shot: "Man Changing Car Battery" by Md Amir Umar (Pexels), Pexels License. Cropped, face and battery label blurred, AR overlay added. Trailer voice: Microsoft en-US-ChristopherNeural; score and sound effects synthesized in code.
  • Logo, code and trailer made at the hackathon. Code is MIT-licensed. Trademarks belong to their owners.

Built With

Share this project:

Updates

Submission history