The Third Eye

Inspiration

The genesis of Third Eye stemmed from observing the stark technological divide in assistive devices. While modern artificial intelligence has reached unprecedented heights, the primary navigational tool for the visually impaired remains the rudimentary white cane. High-end cognitive devices exist in the market, but their exorbitant costs—often exceeding ₹2,00,000—effectively act as a prohibitive "disability tax." The driving force behind this project was the urgent need to democratize machine perception. The goal was to engineer a system that delivers the cognitive power of an elite wearable at a fraction of the cost, ensuring that independent mobility and digital literacy are accessible rights, not exclusive luxuries.

What it does

The system was engineered using a proprietary Hybrid Split-Core Architecture to balance extreme low latency with deep cognitive intelligence.The hardware stack relies on a split-logic configuration. A Raspberry Pi Zero 2 W serves as the primary processing unit, executing a quantized TensorFlow Lite (TFLite) model for offline object detection. To prevent system bottlenecking during heavy asynchronous operations, an Arduino Nano is utilized to manage low-latency hardware interrupts and sensor polling. To guarantee system stability during maximum CPU load (such as simultaneous vision inference and audio transmission), the power architecture is completely decoupled; the Arduino Nano and essential logic components are powered by a separate, dedicated 24000 mAh power bank.The software is structured as a Multi-Threaded Asynchronous Pipeline:The Reflex Engine (Edge AI): Runs continuously offline, calculating spatial hazards. The architecture guarantees a maximum system latency ($L_{max}$) defined by:$$L_{max} = t_{capture} + t_{inference} + t_{alert} \le 50 \text{ ms}$$The Cognitive Engine (Cloud AI): Remains dormant until triggered by a local wake-word engine ("Edith"). Once activated, it captures a frame, encodes it, and queries OpenAI's GPT-4o Vision API to generate a semantic description of the scene, which is then synthesized into human speech.

How we built it

The system was engineered using a proprietary Hybrid Split-Core Architecture to balance extreme low latency with deep cognitive intelligence.The hardware stack relies on a split-logic configuration. A Raspberry Pi Zero 2 W serves as the primary processing unit, executing a quantized TensorFlow Lite (TFLite) model for offline object detection. To prevent system bottlenecking during heavy asynchronous operations, an Arduino Nano is utilized to manage low-latency hardware interrupts and sensor polling. To guarantee system stability during maximum CPU load (such as simultaneous vision inference and audio transmission), the power architecture is completely decoupled; the Arduino Nano and essential logic components are powered by a separate, dedicated 24000 mAh power bank.The software is structured as a Multi-Threaded Asynchronous Pipeline:The Reflex Engine (Edge AI): Runs continuously offline, calculating spatial hazards. The architecture guarantees a maximum system latency ($L_{max}$) defined by:$$L_{max} = t_{capture} + t_{inference} + t_{alert} \le 50 \text{ ms}$$The Cognitive Engine (Cloud AI): Remains dormant until triggered by a local wake-word engine ("Edith"). Once activated, it captures a frame, encodes it, and queries OpenAI's GPT-4o Vision API to generate a semantic description of the scene, which is then synthesized into human speech.

Challenges we ran into

Developing a simultaneous dual-AI system on severely constrained hardware presented significant engineering hurdles.

The most critical challenge was compute and thermal throttling. Pushing simultaneous real-time video inference and audio resampling on a microcontroller with only 512MB of RAM pushed the CPU near its thermal limits. This was mitigated by engineering an intelligent frame-throttling algorithm—processing only the frames necessary for human walking speeds—and aggressively quantizing the TFLite models to run within a safe thermal envelope.

A secondary challenge was network dependency. Cloud-based cognitive features naturally degrade in areas with poor internet connectivity. This necessitated the strict architectural decoupling of the two engines. The system was designed so that the life-saving Reflex Engine remains strictly offline, ensuring physical safety is never compromised by an API timeout.

What we learned

The development journey provided profound, hands-on insights into frugal engineering and edge AI optimization. Extensive knowledge was gained in model quantization—specifically the intricacies of shrinking a neural network to run on minimal hardware without sacrificing critical detection accuracy.

Furthermore, mastering asynchronous Python programming (asyncio and threading) proved essential. Understanding how to manage Mutex locks and shared memory queues was the only way to prevent the cloud-based API calls from blocking the real-time local hazard detection loop, ultimately proving that highly complex AI orchestration can be achieved on highly accessible hardware.