Problem-Solution Fit
Blind and visually impaired people need to know both where they are going and what is directly in their path. A cane can provide valuable information through contact with obstacles, but holding a device like this can make everyday tasks like carrying groceries or holding an umbrella more difficult. In a 2020 navigation study, 64% of the blind participants preferred hands-free devices (Giudice et al., 2020). They noted that holding and operating a phone or opening doors is especially difficult with a cane or guide dog. In another survey accepted at ICCHP 2026, the research found that 26 of 42 visually impaired participants preferred wearables over handheld devices (Bamdad et al., 2026). These studies show the need for these kinds of devices in the visually impaired and blind community.
Another thing we found in our research is that stairs and glass pose a particular hazard for conventional depth sensing devices. Descending stairs can look like open space because the ground drops away instead of forming an obstacle in front of the user, and glass is known to let the sensor’s light pass through which can cause missing readings or measurements of objects behind it. In addition, pedestrians are another challenge because they are constantly moving and are unpredictable. Just because we know where someone is at the current moment doesn't mean we can know where they're going. A study with 12 blind cane users recorded 31 pedestrian contact events, including cases where people failed to move aside or reacted incorrectly to changes in direction (Ohn-Bar et al., 2019). We address all of these hazards in our solution, as we wanted to focus on tailoring this belt specifically for hands-free use and for ensuring safety by making sure the most common hazards are addressed.
Our solution Lantern is a hands-free belt that combines the iPhone’s LiDAR and camera with three custom fine-tuned machine learning models that address the three different navigation challenges we discussed in the problem statement: stairs, glass, and pedestrians. Our stair model identifies the vertical and flat surfaces of steps so Lantern can recognize a staircase as a hazard instead of relying only on obstacle distance. Our glass model combines camera images with depth information to identify glass even when depth readings are missing or unreliable. Our pedestrian detector works with tracking and short-term movement prediction to estimate whether someone is moving into the user’s path. We feed these hazard detections into our obstacle guidance system, which uses three buzzing motors to communicate warnings and directional cues without requiring the user to hold or look at their phone.
We also designed the app so blind and visually impaired users can operate it without needing to look at the screen at all. Siri integration and spoken destination entry let users request navigation by just their voice, while VoiceOver support and large, labeled controls make the interface accessible through a screen reader. We included high-contrast colors to support users with remaining vision. The belt itself is made to look discreet, with a phone holder that can fit the users' iPhone securely. Overall, we spent a lot of time researching and designing our product to address real challenges that blind people face, and we are really proud of how well our product fit the problem we outlined through our research.
References
Bamdad, B. M., Günther, M., & Darvishy, A. (2026). Navigating independence: A survey of visually impaired people’s experiences and needs [Preprint accepted for presentation at ICCHP 2026]. https://digitalcollection.zhaw.ch/server/api/core/bitstreams/4f4a2e72-f037-4d72-9796-64fa7c2ba32b/content
Giudice, N. A., Guenther, B. A., Kaplan, T. M., Anderson, S. M., Knuesel, R. J., & Cioffi, J. F. (2020). Use of an indoor navigation system by sighted and blind travelers: Performance similarities across visual status and age. ACM Transactions on Accessible Computing, 13(3), Article 11. https://doi.org/10.1145/3407191
Ohn-Bar, E., Guerreiro, J., Ahmetovic, D., Kitani, K., & Asakawa, C. (2019). Towards understanding interaction of visually impaired navigators with surrounding pedestrians [Workshop paper]. H-Augmented Workshop, Robotics: Science and Systems. https://eshed1.github.io/papers/peds_RSS.pdf
Accessible Design
The user interface was designed to address the real needs of blind and visually impaired users. Research into accessible design informed our use of strong colour contrast with specific hues and colour combinations, large controls, and a consistent layout. The design also uses Atkinson Hyperlegible, a font developed by the Braille Institute to make individual letters and numbers easier to distinguish for people with low vision. We made tradeoffs between visual styling and ease of use to prioritize accessibility. Together, these choices aim to make information easier to read and controls easier to locate. For user experience, we also integrated apple VoiceOver so users can navigate the interface without needing to interpret the text on the screen at all. Siri was integrated to let users set their route and destination by just their voice, while VoiceOver helped users identify exactly which buttons and controls were being used on the interface. You can find our Figma Demo link included in our submission!
Inspiration
Navigation is a daily challenge for the blind and visually impaired. People who use a cane can face stigma, and keeping one hand occupied can make everyday tasks harder. Tactile paving helps guide the way, but it isn’t available everywhere. We wanted to build something more convenient and discreet that could help users navigate their surroundings. Our idea was a wearable belt that signals to the user through buzzers to warn them about nearby obstacles, keeping users’ hands free.
LANTERN (or just Lantern) stands for LiDAR-Assisted Navigation with Tactile Response and Notification. The acronym represents a light illuminating the path ahead, like the lantern on an anglerfish.
What it does
Lantern combines iPhone LiDAR, on-device machine learning, and a custom belt to detect obstacles and communicate guidance without requiring the user to look at a screen or use a cane.
Here are all the things it can do:
We wanted to focus heavily on safety, since this device would be hands-free. We developed and trained three machine learning models for different types of common hazards.
- Stair detection model: We fine tuned DeepLabV3-ResNet50 using RGB images and depth data to identify the vertical and flat surfaces of each step. This is important because the top of a descending staircase might look harmless but recognizing it as a staircase will help Lantern flag it as a hazard.
- Glass detection model: We built on a pretrained SegFormer-B0 model and added a custom depth branch with a learned gate to combine camera and depth information. This helps it detect glass even when LiDAR gives missing or unreliable readings because sometimes it can miss glass by measuring the objects behind it instead.
- Pedestrian detection and trajectory mapping: Humans can be unpredictable, so having a model map the predicted trajectory of people’s movements helps with hazard prevention. We used pretrained YOLO detection with LiDAR depth to locate people in 3D. We track their positions in ARKit world coordinates and estimate their movement over the next 0.6 seconds, and we also account for the phone’s own movement.
We also included obstacle guidance
- We exported our models to Apple Core ML so they run directly on the iPhone without waiting for a cloud response.
- We used Google’s Navigation SDK for walking directions and Google Places API for destination search. We combined those directions with LiDAR measurements and our hazard detections to decide which way the belt should signal.
- Nearby obstacle warnings take priority over route directions. If sensor data is outdated or there isn’t enough coverage, Lantern stops giving directional cues.
We made a web dashboard to visualize the 3D scans and the machine learning overlays
- We used ARKit LiDAR to reconstruct measured surfaces as a camera-colored 3D point cloud.
- Our dashboard displays model overlays, pedestrian movement trails, depth confidence, navigation decisions, and motor commands.
We built an assistive belt that can be worn discreetly and work with the user’s phone
- We built a belt with a phone attachment and three buzzing motors positioned from left to right.
- An ESP32 receives commands over Bluetooth Low Energy to communicate directional cues and obstacle warnings through different buzzing patterns.
We also made an accessible iOS app that integrates with voice features on the iPhone
- We added high-contrast colors, large labeled controls, VoiceOver support, Siri integration, and spoken destination entry to make it user friendly for blind and visually impaired users.
- Our Google Maps navigation and Google Places search integration lets users request destinations by name or address.
We also focused a lot of Databricks integration for model development and recorded data
- We used GPU notebooks, MLflow, and Unity Catalog to develop our models
- We also implemented the option for users to participate in recording uploads and sending this data to a Lakeflow workflow that stages sensor files and processes them into structured Lakehouse tables so that they can be used for hazard analysis and improving the models
How we built it
For the iPhone app and LiDAR:
We built the app with React Native, Expo, and TypeScript, then wrote a custom Swift module to access ARKit LiDAR, Core ML, and Bluetooth Low Energy. This let us use native iPhone features while keeping the app interface in React Native. We had to run home to grab a Mac and hook it up to a TV as a monitor so we could build and test the iPhone app in Xcode.
We used ARKit’s depth measurements, camera intrinsics, and phone pose to reconstruct measured surfaces in 3D. We also used depth confidence to distinguish reliable measurements from uncertain readings.
For the Machine learning:
We used Databricks GPU notebooks and PyTorch to fine tune our stair and glass segmentation models. We exported the models to Apple Core ML and matched the iPhone’s input preprocessing to our training pipelines. The segmentation models produce image masks, which we align with the matching LiDAR frame to label measured 3D points. For pedestrians, we track positions in ARKit world coordinates and estimate short-term movement while accounting for the phone’s motion.
For the Navigation and voice control:
We integrated Google’s Navigation SDK for walking directions and Google Places API for destination search. We combined route directions with local LiDAR clearance and model-detected hazards to choose directional cues, with immediate obstacle warnings taking priority.
For the Assistive Belt and electronics:
We wrote the ESP32 firmware in Arduino C++ and connected it to the iPhone over Bluetooth Low Energy. The phone sends a versioned command packet that controls the belt’s left, centre, and right buzzing outputs. We programmed separate patterns for directional guidance and obstacle warnings, and the firmware clears its outputs if the connection drops or commands stop arriving.
For the web dashboard:
We built the dashboard with React, TypeScript, Vite, and Tailwind CSS, with WebGL rendering for the point cloud. It receives opt-in telemetry from the phone and displays live scans, model overlays, pedestrian trails, sensor confidence, and belt commands. We also added simulation scenarios so we could develop and test the interface without needing the phone or belt connected.
For the Databricks part:
We used Unity Catalog Volumes to store datasets and model artifacts, MLflow to track experiments, and the Unity Catalog Model Registry to manage model versions.
For recorded walks, we implemented an upload queue that saves ML recording ZIPs on the phone and retries failed uploads (users can opt-in to do this). Our backend sends them through the Databricks Files API while keeping Databricks credentials off the iPhone. We configured a Lakeflow Job with two tasks to validate and extract uploaded ZIPs, then run our Apache Spark pipeline. The pipeline organizes the recordings into six Delta Lake tables following the Bronze, Silver, and Gold Medallion Architecture that Databricks recommends.
Databricks Integration
- We used Databricks GPU notebooks to train and fine tune our stair and glass segmentation models without needing local GPU hardware
- We used Unity Catalog Volumes to keep our datasets, checkpoints, config files, and exported models together
- We used MLflow to compare training runs and fine tuning experiments, then we published our models to the Unity Catalog Model Registry to manage their versions
- We used the Databricks Files API to upload walking recordings through our backend while keeping Databricks credentials off the iPhone
- We configured Lakeflow Jobs to automatically stage new recording uploads and run our Lakehouse pipeline
- We used Apache Spark to process sensor recordings, check timestamps and file references, and analyze hazard detections and pedestrian motion
- We used Delta Lake to organize six tables following Databricks’ recommended Medallion Architecture:
- Bronze stores raw recording metadata and removes duplicates
- Silver checks sensor files and timestamp alignment
- Gold produces hazard events, pedestrian tracks, walking session summaries, and review candidates
Challenges we ran into
Our first big challenge was hardware. We originally planned to use vibration motors, but our same day delivery didn’t arrive. We decided to use three buzzers as temporary placeholders to demonstrate left, front, and right cues, with rapid beeping for obstacle alerts.
Making sense of sensor data was another challenge. A missing depth reading does not mean the path is clear. Glass made this especially obvious: a model might recognize glass in an image while LiDAR measures the wall behind it. We had to be careful about what the system could recognize and what it could reliably measure. We trained a machine learning model to address this hazard.
Connecting the app, the hardware and our custom machine learning models within 24 hours was also a major challenge. We had too many integration meetings throughout this hackathon, but it worked out!
Accomplishments that we're proud of
We’re proud of putting Lantern on a physical iPhone and streaming live ARKit data to the dashboard. The phone is able to detect a person and show the detection alongside the depth readings on the dashboard.
We integrated three Core ML models, registered them in Unity Catalog, and built a structured pipeline for reviewing recordings and supporting future model improvements. We’re also proud of the user interface. We used existing research and resources made for people with visual impairments to guide our design choices.
What we learned
We learned to keep accessibility in mind throughout the entire project and design even the small details around the people we were building for
We learned to keep working and moving forward even though some parts weren't ready. We tested the app and dashboard with simulated inputs while we waited for the hardware and models.
Most importantly, we learned that a working prototype is a starting point. Understanding whether Lantern is useful will require testing with the community that we’re designing for.
What's next for Lantern
In the future, we would like to build and test the belt with vibration motors. We would also like to consult the blind and visually impaired community to validate our design decisions and gather feedback on what to change and how we can improve our design
In addition to this, we plan to process walking recordings through Databricks, review challenging examples, and use them to improve the models.
We will continue to focus on stair and glass detection, and how clearly the belt communicates to the user. Lantern is still an assistive prototype, and making it dependable for everyday use will take more testing and user feedback.
Built With
- css
- databricks
- javascript
- python
- swift
- typescript
