StreetSmart

Car detection is useful for preventing accidents, especially against aggressive/unsafe drivers.
Traffic light detection works on both pedestrian and vehicular traffic lights.
Detections are still accurate from long distances.

Inspiration

We've noticed a lack of free/low-cost technologies aimed at helping blind/low-vision (BLV) users navigate urban areas. In NYC, traveling requires crossing many intersections and through a lot of traffic, which presents a great challenge for BVI users who are unfamiliar with the area, therefore we decided to build an app that BLV users detect traffic light signals and incoming cars through a phone camera.

What it does

The app uses real-time object detection to identify traffic lights and vehicles through the phone camera. When a traffic light or an oncoming car is detected, the system delivers audible alerts via custom AI-generated text-to-speech. Users can also plan routes and set destinations, receiving personalized audio navigation guidance throughout their journey.

How we built it

We utilized PyTorch-based YOLO (You Only Look Once) object detection models to identify and classify traffic lights and vehicles in real time. On the frontend, we employed the JavaScript Media Capture and Streams API in conjunction with the Canvas API to capture video frames and continuously transmit them to the Flask backend for inference. Additionally, we integrated ElevenLabs' text-to-speech service to deliver audible alerts to users, providing real-time feedback on detected objects. Also, we've incorporated the Google Maps API to let users plan routes. Extracting the directional route feature data, we use the user's current location to generate turn by turn Text-To-Speech as users are walking to their destination.