Inspiration: Continuous vision agents exist natively on High Price headsets or deep OS-level apps (like ChatGPT or Pixel's Gemini Live). Why? Because running continuous WebRTC, microphone polling, and TTS simultaneously in a standard mobile browser usually crashes the UI or melts the battery. We wanted to democratize this experience. Our inspiration was a pure engineering challenge: could we build an autonomous, zero-install, browser-based visual auditor without hitting the hardware limits of Mobile Safari/Chrome? OpticFlow is the answer.
What it does: OpticFlow turns any mobile browser into an AR HUD. Once initialized, you tell it what you want to do (e.g., "help me descale this coffee machine"). It fetches the manual and gives you step one. You put your phone down and start working. OpticFlow silently captures frames every 8 seconds, evaluating your physical progress. If you do it right, it praises you and gives the next step. If you do it wrong, it corrects you. If you walk away, it automatically times out and shuts down.
How we built it: We built the backend using Python, Flask, and the new google.genai SDK, deployed statelessly on Google Cloud Run. The core engine is Vertex AI's Gemini 2.5 Flash. The hardest engineering challenge was managing the browser's hardware limitations. We wrote a custom JavaScript Mutex controller to prevent the continuous camera loop, the microphone, and the TTS engine from colliding. We also implemented a "Bifocal" resolution system to save bandwidth during background checks, only using high-res images when OCR is strictly required.By managing these constraints, we achieved an enterprise-grade native app experience entirely inside the web DOM.
Log in or sign up for Devpost to join the conversation.