SecondSight

This is how it works as you can see in the right side... Every 8 secs it analyses the scene,describes the safety hazards. also accept voice

Inspiration## 💡 Inspiration

For 2.2 billion people with vision impairment, existing tools lack contextual, real-time awareness. They can read a static sign but fail to spot an immediate, silent hazard or guide complex navigation. We set out to move beyond simple OCR (Optical Character Recognition) to build a tool that delivers Immediate, Actionable Safety Intelligence for true independence.

What it does

SecondSight is a real-time, voice-activated "digital vision cortex." It works exclusively on user-initiated commands to deliver maximum safety and resource efficiency. Guardian Mode (Live Snapshot): Instead of simple photo upload, the app uses the front-facing camera in Live Snapshot mode. Analysis is performed only when the user speaks a command or taps the 'Ask Question' button, prioritizing quota management while providing instantaneous results. Contextual Safety Intelligence: It identifies and synthesizes: Hazards/Allergens: Reading medication labels, warning of 'Wet floor signs,' or identifying food allergens (e.g., "The second pill bottle is for a 5mg dose; no gluten warnings seen."). Action-Oriented Voice Output: Using the Web Speech API, the AI's analysis is converted into concise, spoken instructions tailored for navigation ("Stop. Steps descending immediately ahead. Go left."). Twist - Conversational Recall: By taking a fresh snapshot with every user question, Gemini maintains context not only from the current visual scene but also from the previous conversational turn, making Q&A seamless.

How we built it

We achieved high complexity by tightly integrating advanced front-end capabilities with Google's latest model, while aggressively coding defensively against network/quota constraints. Core Technology - Full Front-End Stack: Frontend: Built entirely in React/Vite for a responsive, installable PWA experience, requiring no backend for the core functionality. Real-Time Snapshot Engine (HIGH COMPLEXITY): We developed a frame-grabbing engine that replaces file upload. This system uses a hidden HTML5 element, renders to a temporary element, and compresses the result down to 30% JPEG quality (to fight the new latency/quota issues), creating a Base64 image payload ready for Gemini on demand. AI Engine & Defenses: Gemini 2.5 Flash Integration: Selected and implemented the multimodal Gemini 2.5 Flash for the fastest, most economical vision analysis. Anti-Quota Throttling: After hitting severe quota limits, we surgically removed the automatic continuous scanning loop and restructured the app to fire a single API call only upon explicit user request (Voice Command or Button Tap). This ensured a stable demo with perfect response times. UI/UX Refinement: We moved beyond a simple chat layout to a high-contrast "Guardian Console" featuring a dedicated live-feedback status pane to provide users with crucial state cues (Listening, Thinking, Safe)

Challenges we ran into

Quota Crisis (CRITICAL HACKATHON LESSON): Our initial continuous real-time analysis consumed the quota for several of our keys. We overcame this by switching to a User-Initiated Snapshot model and using a local code timer to force-kill stuck API requests, ensuring the UI never permanently freezes—a robust failure-handling technique. Payload Latency: Even with the "Flash" model, transferring high-resolution Base64 data took too long in sandbox environments. This was solved by aggressively reducing the canvas rendering quality to the minimal viable level (0.3 JPEG quality), dramatically reducing payload size and improving real-time feel

Accomplishments that we're proud of

Functional Prototype: We built a fully working "Snap-to-Speech" pipeline in under 4 hours. Smart Synthesis: We successfully synced the AI's text generation with the browser's native Text-to-Speech engine, creating a seamless conversational experience without needing expensive 3rd party audio APIs

What we learned

Multimodality is powerful: We learned that passing images to an LLM isn't just about "captioning." The model can actually "reason" about physical space (e.g., knowing that a chair in a hallway is a trip hazard, not just furniture). Prompt Engineering = Logic: We realized that writing the English prompt for Gemini was just as important as writing the JavaScript logic. Quota-Resilient Full Pipeline: Building a functional multi-modal, voice-driven app that reliably navigates API constraints. Pure Front-End Engine: Achieving "real-time" multi-modal image analysis without a traditional server, relying on native browser and Gemini APIs alone. Full Speech Cycle: Seamlessly linking the microphone's input (Web Speech Recognition) to the AI, and then to the text-to-speech output (Web Speech Synthesis).

What's next for SecondSight

Backend Integration (MongoDB/Full-Stack): Adding a secure Node.js backend to enable History Review (saving the sessions to MongoDB) and personalized persistent user settings. Micro-Interaction: Implementing haptic feedback patterns (one buzz for "safe," two quick buzzes for "alert") to communicate critical non-text data immediately.