Inspiration
Proactive Lab Partner is a real-time, vision-powered desktop application acting as an AI co-pilot for hardware engineering. Using a standard webcam, it continuously monitors your physical circuit board or Arduino setup. If it detects a wiring hazard, missing component, or potential short circuit, it immediately interrupts you with a spoken audio warning, allowing you to fix the mistake before applying power.
Key Features Real-Time Vision Analysis: Captures and analyzes live camera frames of physical hardware setups.
Instant Audio Alerts: Utilizes browser-native Text-to-Speech (TTS) to deliver hands-free, spoken warnings the moment a hazard is detected.
Manual & Continuous Modes: Allows users to trigger a manual scan for specific checks or run continuous polling for active monitoring.
Native Desktop Experience: Packaged as a standalone Windows .exe application, removing the need for terminal commands or browser tabs.
The Tech Stack AI Engine: Google Gemini API (gemini-flash-latest) for high-speed, multimodal vision-language processing and JSON structuring.
Backend: Python, FastAPI, and Uvicorn for handling local server requests and image payload decoding.
Frontend: Vanilla HTML, CSS, and JavaScript, leveraging the native Web MediaDevices API (for webcam capture) and Web Speech API (for voice synthesis).
Packaging & Distribution: PyInstaller (for executable compilation), Pywebview (for native window rendering), and Inno Setup (for the Windows installation wizard).
How We Built It We started by testing the Gemini API's multimodal capabilities in Google Colab to see if it could accurately identify breadboard components and wiring errors from raw image frames. Once we validated the system prompt and locked in a strict JSON response schema, we migrated the architecture to a local Visual Studio Code environment. We built a FastAPI backend to route base64 image strings to Google's servers and a sleek HTML/JS frontend to handle the webcam feed. Finally, we compiled the entire stack into a single, distributable Windows setup file using PyInstaller and Inno Setup.
Log in or sign up for Devpost to join the conversation.