Buddy

How Buddy came to be

Fahim saw it firsthand in Boston: a man with a white cane hesitated at a crosswalk, unsure when it was safe to step forward. In broad daylight, on a seemingly normal street, he was completely vulnerable. The image stayed with Fahim – how something as simple as crossing a street could feel like a life-or-death decision.

Apurvaa experienced something similar on campus. While managing a building, she witnessed a visually impaired person nearly trip over an unexpected obstacle. He came up to her, frustrated, upset, and scared. A single misstep, and it could have been catastrophic.

Amanda’s story is personal. Born prematurely at 25 weeks, she has -14 myopia and is still legally blind even after LASIK. Not everyone is fortunate enough to have access to corrective surgery, glasses, or contacts. Without those 3 things, something as simple as walking down a street would be impossible for her. She constantly thinks about what life would have been like navigating the world completely without sight.

Yan’s family has battled vision impairment for generations, and he has felt the weight of it firsthand. He’s seen how something as simple as a misplaced object, a missing tool, or a small oversight can turn everyday life into a maze of danger and uncertainty. Safety, independence, and even confidence all hinge on these tiny details, and the consequences of neglect are all too real.

Together, these experiences brought us together, united by a shared understanding of vulnerability and a determination to build something that could protect everyone. That’s how Buddy was born.

What is Buddy?

Buddy is a mobile safety intelligence system designed to help pedestrians understand their surroundings in real time. It acts as a digital second set of eyes, scanning the environment and identifying potential threats before they become dangerous. By combining live camera input, depth estimation, and AI-driven audio feedback, Buddy turns complex street environments into fast, intuitive safety signals. Buddy recognizes crosswalks, traffic lights, moving vehicles, street signs, and unexpected obstacles, but it goes beyond just normal object detection. It interprets movement, speed, and distance to understand risk in context. When danger is detected, Buddy delivers instant voice alerts, allowing users to respond immediately without looking at their screen. The result is a seamless, background safety system designed for real-world movement.

Fun fact: Buddy was originally built to assist people with visual impairments – just like a guide dog, it’s there to watch your back, alert you to danger, and keep you moving safely. But the truth is, Buddy isn’t just helpful. It’s essential. Anyone who walks on streets, sidewalks, or crosswalks needs Buddy.

Why it matters

Walking shouldn’t feel dangerous. Yet every day, pedestrians face risks that shouldn’t exist.

Drivers speed past stop signs and crosswalks, ignoring the very systems meant to protect people. In Pleasant Grove, Utah, just two days ago, a high schooler was injured in a crosswalk in broad daylight after a driver was speeding, not stopping for stop signs. Last week, 19-year-old Ashlee Reeves, a SUNY Cobleskill student, was fatally struck while walking across a barely-lit street. You could be the most careful, physically fit, alert person – but one mistake by a driver, or one blind spot, and it can all be over in an instant.

But danger isn’t just external. It’s also the world we’ve created for ourselves. We live in a digital age where almost everyone is distracted. Phones, headphones, music, podcasts, notifications – they pull our attention away from the simplest, most important thing: staying alive while walking. This isn’t just our fault; it’s human nature. But it’s exactly what Buddy is built to fix. By leveraging the phone’s camera, Buddy can even detect when you’re looking down at your device and remind you to stay present.

According to the AAA Foundation for Traffic Safety, pedestrians distracted by their phones are four times more likely to be involved in a collision. The National Safety Council found distracted walking factored into 2,500 deaths in 2021 alone. In 2023, a pedestrian was killed every 72 minutes in the United States. More than 3,800 people died on U.S. roads in pedestrian crashes that year – a stark reminder that new advancement in technology in the automotive industry has led to higher speeds and bigger, quieter vehicles, making streets deadlier than ever. Audio is extremely important to those who are visually impaired. Marston and Golledge studied street crossings by blind pedestrians in 2000 at intersections with visible but not audible pedestrian signals, such as the sounds of engines revving or tires. Almost half of the participants’ crossings occurred when the DON’T WALK signal was on. Furthermore, about half of the participants made crossings that were deemed unsafe on each of their 20 attempts to cross the street.

Even policy struggles to keep up. While the Biden administration provided modest funding to improve pedestrian safety, efforts to protect pedestrians have been inconsistent under the Trump administration. A federal official told Boston’s transportation agency in September that the administration was pulling funds for a project it deemed “hostile to motor vehicles,” according to correspondence obtained by The Washington Post.

However, distracted walking isn’t just a U.S. problem. In Accra, Ghana’s central business district, a major economic hub with heavy pedestrian traffic, researchers studied 400 commuters to understand what distracts people while walking. Listening to music on a mobile phone emerged as the major distraction: 79% of respondents ranked it as their most common distractive activity. Making or receiving phone calls and conversing with other people while walking followed. Browsing the internet on mobile phones ranked 4th, and was widespread among those aged 9-24 and 27-42.

Every day, people are crossing streets they think are safe. And every day, Buddy can make the difference between life and tragedy.

How Buddy works

Buddy integrates real-time computer vision, depth estimation, and AI reasoning to provide continuous pedestrian safety.

Live Video Capture: Users record a brief, five-second video of their surroundings using their phone’s rear camera. This short clip provides enough visual context for Buddy to understand nearby objects, movement, and layout, harnessing Gemini AI’s API to detect fast-moving objects and sudden hazards.

Depth Perception: Buddy applies Gemini to calculate relative distances to detected objects. By combining bounding box coordinates with depth maps, Buddy distinguishes between stationary and approaching hazards, enabling predictive risk assessment.

Contextual Risk Scoring: Gemini synthesizes spatial, motion, and environmental data to generate a real-time risk score. This score determines which events require immediate notification versus passive monitoring.

Voice Feedback: Gemini prioritizes the most critical safety messages, which are then delivered through ElevenLabs’ emotionally expressive voice synthesis. The voice is calm but urgent, providing clear instructions that reduce panic while ensuring rapid response.

What's next for Buddy?

Buddy is just getting started. Today, it helps users better understand their surroundings through fast, on-demand visual analysis, but our vision goes far beyond that. In the future, Buddy could support continuous live video analysis, allowing real-time awareness without requiring manual recording. And we hope to expand audio alerts to haptic as well. We envision Buddy extending to wearable devices like smartwatches or bracelets that can deliver vibration alerts directly to your wrist, making safety truly hands-free. By integrating LiDAR sensors, Buddy could detect obstacles and moving hazards even without relying on the camera, enabling awareness in low-visibility situations or when the phone isn’t perfectly positioned. Future updates could include predictive hazard modeling using AI to anticipate risks before they appear and sensor fusion that combines GPS, accelerometer, LiDAR, and camera data for a complete understanding of your environment. We also imagine community-driven safety features, where anonymized data helps cities identify dangerous intersections, alert pedestrians in real-time, and improve urban planning and policymaking.