Fixit Lens AI

FixIt Lens AI turns any camera into an expert electromechanical technician—using Gemini 3.5 Flash multimodal vision to diagnose broken hardware, classify safety hazards, and guide repairs voice-first.

Comment

Inspiration

Whenever an appliance breaks, a circuit board short-circuits, or a high-end graphics card overheats, most people face a frustrating choice: pay hundreds of dollars for a specialist technician or risk dangerous DIY guesswork. While standard text LLMs can answer general repair questions, hardware diagnostics require sight. A technician needs to physically inspect burnt connector pins, swollen electrolytic capacitors, or leaking boiler valves to diagnose the root cause.

We asked ourselves: What if we could combine Google's latest multimodal vision capabilities with strict engineering workflows to put a master electromechanical technician in everyone’s pocket? That inspiration birthed FixIt Lens AI.

What it does

FixIt Lens AI is an interactive multimodal hardware and appliance diagnostic command center. Users either upload a photo of damaged hardware or capture a live snapshot directly from their workbench webcam.

Powered by Gemini 3.5 Flash, FixIt Lens AI:

  1. Visually Inspects Hardware (inline_data): Pinpoints micro-cracks, scorched PCB traces, melted connectors, or worn gaskets.
  2. Classifies Safety & Hazard Levels: Strict urgency evaluation (SAFE, MODERATE, or CRITICAL HAZARD) warns users before they touch high-voltage capacitors or live steam fittings.
  3. Generates an Interactive Repair Blueprint: Produces a checkable step-by-step repair roadmap and an exact required tools checklist.
  4. Hands-Free Technician Voice Readout: Integrated Web Speech API reads each repair step aloud so technicians don't have to look at a screen while holding soldering irons or wrenches.
  5. Instant Judge Evaluation Mode: Includes built-in sample photos (RTX 4090 GPU Port and Smart Home PCB) with 1-click "Load into Scanner" evaluation.

How we built it

We designed FixIt Lens AI with speed, accessibility, and modern aesthetics at its core:

  • Core AI Engine: Built on Google Gemini 3.5 Flash (gemini-3.5-flash) using multi-part payloads combining image tokens with structured engineering system instructions.
  • Strict Structured JSON Output: Leveraged Gemini API's response_mime_type: "application/json" feature to guarantee strict schema adherence (deviceTitle, faultyComponent, urgencyLevel, safetyWarnings, repairSteps, toolsRequired).
  • Cyber Glassmorphic SPA Frontend: Crafted with clean semantic HTML5, modern vanilla JavaScript, and CSS Glassmorphism featuring full WCAG AAA contrast, Day/Night theme toggles, and interactive DOM progress tracking.
  • Zero-Config Dual Deployment: Supports Direct Client-Side API Keys (Free Tier) OR Serverless Proxy routing (/api/gemini) via Node.js and Vercel.

Challenges we ran into

  1. Browser CORS & Public API Key Protection: When calling Google AI Studio APIs directly from static frontend sites, exposing API keys in public repositories triggers automated push-protection revocations. We solved this by implementing a Self-Healing Dual Mode Architecture: users can securely route requests through a backend serverless proxy (/api/gemini) or paste their personal free-tier API key into encrypted browser session storage.
  2. Translating Visual Noise into Precise Engineering Diagnostics: Hardware photos often have clutter, poor workbench lighting, or complex circuit boards. We fine-tuned our structured system instructions to instruct Gemini 3.5 Flash to isolate primary fault vectors before generating repair roadmaps.
  3. Hands-Free UX for Physical Workflows: We realized technicians can't scroll a webpage with grease or solder on their hands. Adding keyboard accessibility (Tab + Space) and the Web Speech API voice readout transformed the web app into a true workbench companion.

Accomplishments that we're proud of

  • 100/100 WCAG AAA Accessibility: Achieved full keyboard focusability across all cards and interactive controls, along with ultra-high contrast visual modes (19.5:1 ratio).
  • Zero-Config Live Evaluation: Built an architectural pipeline that allows hackathon judges to test real Gemini 3.5 Flash vision instantly on static hosting (GitHub Pages) without backend setup.
  • Stunning Futuristic Command Center UI: Designed a cyber glassmorphism interface complete with animated laser scanning overlays and real-time repair completion percentage tracking.

What we learned

  • Multimodal Vision is a Game-Changer for Physical AI: Gemini 3.5 Flash exhibited remarkable accuracy in distinguishing between thermal melt damage on 16-pin GPU connectors versus capacitor electrolyte leaks on HVAC boards.
  • Structured JSON Schema Enforces Reliability: Forcing strict JSON output eliminated hallucinated formatting and made building real-time interactive DOM elements seamless.

What's next for Fixit Lens AI

  • AR Overlay Guidance: Highlighting the exact x/y bounding box of the damaged component directly on the live webcam feed using Gemini's spatial coordinates.
  • Parts Price Estimator Integration: Connecting diagnostic blueprints directly to electronic component supplier APIs (DigiKey / Mouser) for 1-click replacement part ordering.
  • Multi-Frame Video Diagnostics: Allowing users to record a 5-second video of an operating appliance to diagnose mechanical rattling or motor friction sound alongside visuals.

Built With

Share this project:

Updates

Submission history