FixerOS

Inspiration

We’ve all been there—a leaky faucet at midnight, a flickering AC in peak summer, or a car that just won't start. The frustration isn't just the breakdown; it's the information gap. Customers don't know what's wrong (and fear being overcharged), while honest technicians struggle to give accurate quotes without seeing the issue first. We wanted to bridge this gap using Multimodal AI. Inspired by the idea of an "Industrial Jarvis" for the everyday home, we built FixerOS to democratize technical expertise. Our goal? To turn every smartphone camera into an expert diagnostic tool and create a trust-based marketplace powered by Gemini 3.0.

What it does

FixerOS is a dual-sided marketplace (Uber for Repairs) that leverages Gemini 3.0 Flash at every step of the repair lifecycle:

1. For Customers (The "Smart Triage"):

   • Fix-It Bot: An intelligent conversational assistant that diagnoses problems via natural language before a meaningful booking is even made.
   • Smart Scan: Users simply snap a photo of the broken item. Gemini’s multimodal vision automatically detects the appliance, identifies the likely fault (e.g., "Burnt PCB"), assigns urgency, and even estimates a fair budget.

2. For Fixers (The "AR Copilot"):

   • Live Vision Mode: An augmented reality interface where the camera continuously scans the environment using Gemini 3.0, providing real-time labels and diagnostics without pressing a button.
   • AI Technical Breakdown: Before accepting a job, fixers get an AI-generated reasoning report listing required tools, safety warnings, and estimated repair time based on the customer's photos.

How we built it

We built FixerOS as a high-performance PWA (Progressive Web App) to ensure accessibility across all devices.

• Core Stack: React (Vite), TypeScript, TailwindCSS for a futuristic "Cyber-Glass" UI.
• Backend: Firebase (Firestore, Auth) for real-time geospatial queries (locating fixers within 10km).
• The Brain (Gemini 3.0 Flash):
  • We used gemini-3.0-flash for its incredible speed and multimodal capabilities.
  • Vision Pipeline: We capture video frames from navigator.mediaDevices and send them to the API. To achieve "Live Vision," we implemented an optimized polling loop that balances latency with API quotas.
  • Structured Reasoning: We heavily utilized Gemini's ability to output structured JSON to power our UI components (e.g., converting a raw image into a detailed JobAnalysis object with confidence scores).

Challenges we ran into

• Real-time Latency: Making the AR experience feel "live" was tough. Initially, sending high-res images to the AI caused lag. We optimized this by compressing frames and implementing a smart interval system that only analyzes when the camera is steady.
• Structured Output Stability: While Gemini is powerful, ensuring it always returned valid JSON for our app to parse was a challenge. We refined our system prompts significantly to enforce strict schema adherence for things like tool lists and budgets.
• Dual-Role Logic: Managing state for a user who might act as both a Customer and a Fixer required a robust global context system.

Accomplishments that we're proud of

• Hands-Free AR: Successfully implementing the "Live Vision" mode where the AI "sees" and labels objects in real-time. It feels like magic.
• End-to-End AI Flow: We didn't just bolt on a chatbot. AI is intrinsic to the workflow—from the moment a user takes a photo to the moment a fixer gets their safety warning.
• Hackathon Compliance: We fully transitioned our legacy codebase to gemini-3.0-flash and built entirely new features (Bot & Smart Scan) during the hackathon window.

What we learned

• Multimodal is the Future: Text-only inputs are insufficient for real-world tasks like repairs. The ability to "show" the AI the problem changed everything.
• User Trust: AI doesn't perfect the repair, but it perfects the trust. When Gemini suggests a price range, customers feel more confident booking a stranger.
• PWA Potential: You don't always need a native app to deliver powerful AR experiences; the modern web platform is incredibly capable.

What's next for FixerOS

• IoT Integration: Connecting directly to smart appliances to predict failures before they happen.
• Video-Call Supervision: Enabling "Senior Technicians" to guide junior fixers via live video, with Gemini summarizing the call and highlighting key instructions.
• Verified "Pro" Badge: Using AI to analyze completed job photos to automatically rate and verify a fixer's quality of work.

Built With

• firebase-hosting
• framer-motion
• gemini-2.0-flash
• gemini-3-flash-preview
• react
• tailwind-css
• vite
• web-speech-api-(stt-&-tts)