Inspiration
Getting around with vision loss can mean relying on a cane, a guide dog, or a phone app. Each can help, but it can still be hard to spot nearby hazards in a busy hallway, an unfamiliar building, or a changing crowd. We wanted to explore how a smartphone’s sensors and AI could help fill that gap.
Our idea was to make a navigation assistant that does more than give directions. It should interpret the space around the user, offer useful guidance, and make it easier to bring in a caregiver when needed.
Hadar combines iPhone camera and LiDAR sensing with spoken and haptic guidance, plus optional caregiver support.
What it does
Hadar is designed to help people with vision loss understand nearby surroundings while they move.
The iPhone camera and LiDAR capture visual and depth information. Hadar turns depth readings into a simplified map of open and blocked areas, then plans a nearby route and updates its guidance as the user moves.
Users can receive spoken directions, ask questions about the scene, and start a caregiver session. A caregiver can use a web viewer to follow session status and updates from the app.
How we built it
We built Hadar for iPhone using Swift, SwiftUI, and Xcode. The project brings together:
ARKit and LiDAR to capture depth and map nearby space
The iPhone camera to provide visual scene input
Speech and haptics to communicate guidance without requiring the user to watch the screen
Gemini to interpret scenes and answer questions
Firebase and Firestore to support conversation and caregiver session data
A web viewer built with HTML, CSS, and JavaScript for caregivers
For route planning, Hadar converts depth readings into an occupancy grid. The grid marks areas as open or blocked, and a path-planning algorithm selects a route through open space. As the sensor readings change, the app can update its guidance.
Challenges we ran into
Caregiver support was one of the more complex parts of the project. We explored using Daily.co for live video, but getting the video to display consistently in the caregiver viewer was difficult. Streaming high-quality video placed too much demand on the device and connection.
Reducing the stream’s frame rate and quality helped improve stability. The experience also underscored how much accessibility depends on clear, dependable feedback.
What we learned
Working on Hadar helped us learn how to:
Combine camera and LiDAR input
Build occupancy maps from depth data
Apply AI to scene interpretation
Design with accessibility and trust in mind
Connect mobile apps with backend services
Bring voice, haptics, and navigation together
We also saw that even small delays or confusing cues can matter. Accessibility features need careful design and testing in real conditions.
What's next for Hadar
We hope to keep developing Hadar by:
Improving outdoor navigation
Adding GPS-based routing
Better detecting moving obstacles
Expanding caregiver and emergency features
Testing with more users in everyday settings
We want Hadar to help make navigation support more informative and accessible for people with vision loss.
Log in or sign up for Devpost to join the conversation.