Shaded Sheriffs

With Shaded Sheriffs, we enable vision by description. Our smart glasses capture the world before you, using AI to speak it aloud and turn visual barriers into spoken independence.

Comment

Inspiration

The physical world communicates through silent, visual cues—like signage or a handwritten restaurant menu—creating profound daily barriers for the visually impaired. We were inspired to smash through these barriers because true independence starts with understanding your immediate surroundings. With Shaded Sheriffs, we enable vision by description.

What it does

Shaded Sheriffs is a wearable ecosystem providing instant spatial context and safety:

  1. Smart Glasses: A camera captures the wearer's exact view. Generative AI translates this image into a concise, spoken description using a gritty "Sheriff" voice.

  2. Remote Dashboard: A Wild West-themed web UI displays live "Dispatch Records" (analyzed photos and text) so caregivers can remotely monitor and verify the wearer's surroundings.

  3. Ultrasound Belt: A standalone haptic belt uses an ultrasonic sensor to detect immediate physical obstacles, instantly buzzing to prevent collisions.

How we built it

  • Hardware: An ESP32-CAM powers the glasses, while an ESP32 Feather and Arduino Uno drive the ultrasound belt.

  • Backend & AI: A Python Flask server receives Base64 encoded images from the glasses and privately routes them to the Google Gemini 3 Flash Preview API for high-speed, multimodal vision inference.

  • Frontend: A custom HTML/JS dashboard uses the browser's native window.speechSynthesis API to handle real-time Text-to-Speech without backend audio latency.

Challenges we ran into

Our biggest hurdle was achieving consistent photo capture with the notoriously finicky ESP32-CAM. Initial breadboard testing caused severe electrical interference, resulting in ghost frames and corrupted Base64 data streams. We pivoted in the final hours, stripping the breadboard and soldering the critical connections to ensure the clean signal transfer needed for a reliable demo.

Accomplishments that we're proud of

  1. Actuating the ESP32-CAM and successfully streaming images over local Wi-Fi for the first time.

  2. Implementing a Generative AI model (Gemini API), specifically leveraging its multimodal vision capabilities.

  3. Building a cohesive, interactive UI that seamlessly bridges physical hardware and web software.

What we learned

We learned how to integrate cutting-edge GenAI models into physical hardware, which required optimizing Base64 data encoding and managing network latency. We also leveled up our front-end skills, utilizing Python Flask to build responsive microservices bridging the gap between C++ and JavaScript.

What's next for Shaded Sheriffs

  1. Physiological Monitoring: Integrating biometric sensors to track the wearer's heart rate and skin temperature, adding vital safety data to the caregiver's dashboard.

  2. Native Mobile App: Converting our web UI into a dedicated iOS/Android application to make remote monitoring accessible for caregivers anywhere, not just on a local network.

  3. Detection of Movements: Convert instant shots to one real time video that can be used to infer motion and expression.

Built With

Share this project:

Updates