Inspiration
For blind individuals, navigating the world on an everyday basis can be challenging. Unlike sighted individuals, the blind must rely on senses other than vision to build mental maps of their surroundings, such as tactile or auditory cues. This makes it difficult for them to navigate through unfamiliar surroundings; moreover, it is extremely challenging to navigate back to a previous location of interest in complex environments. This gave us the inspiration to develop ARCane, a smart walking cane that enables real-time mapping of surroundings, memory of previously visited locations, and the ability to upload mapped locations to the cloud to share with any user. By integrating LiDAR sensors and a camera directly in the walking cane for minimal hassle, we aimed to tackle these issues by leveraging the power of augmented reality, computer vision, and natural language processing!
What it does
Real-time mapping: When a user enters a new location or room, ARCane starts building a real-time map of the area: using computer vision, it identifies surrounding objects and saves them in memory as they keep exploring the room. They can ask, "What's in front of me?" and the system will respond in natural language with the names of the objects in their surroundings and where the objects are with respect to the user (using LiDAR to estimate distance). For example, "There is a chair 2 meters in front of you and wall 3 feet to the right of you".
Waypoint navigation: The user can save their current location, travel elsewhere in the room, and then navigate back to that location using haptic guidance and voice commands ("turn right", "turn left", etc.) to nudge the user in which direction to travel.
Persisted and open-source room mappings: When a user completely maps a room and its objects, their mapping is saved to the cloud, allowing them to have memory of the room and its surroundings the next time they enter it. In addition, the individual has the option to upload their mapping to the cloud to allow other blind individuals who enter the same room to automatically access the shared mapping.
How we built it
Using Swift and XCode, we built an iOS app that leverages the native iOS hardware API to provide LiDAR, haptics, and augmented reality functionality. In addition, we used a computer vision model for identifying the names of objects in the individual's surroundings.
Challenges we ran into
One of the more difficult aspects of this project was determining how to use haptics to navigate the user to their saved waypoints. At first, we wanted to program a small ESP32 microcontroller to control the vibration of several vibration motors depending on the direction the user needed to go towards. However, neither of us had much experience with hardware, and after a lot of time spent debugging, we ultimately decided to use the iPhone's own haptic vibration motors. We found that by encoding custom vibration patterns to represent each direction of navigation, we were able to effectively integrate the haptic feedback system for navigation.
Accomplishments that we're proud of
We are very proud of the fact that we were able to develop something that addresses the needs of the blind community. We are also happy that we were able to do leverage native iPhone features such as haptics, LiDAR, and speech, as it is something that we have never done previously.
What we learned
Neither of us had any experience with XCode or Swift before; we learned how to set up an iOS app, write code to access Apple's hardware APIs, and test the app in real-time. In addition, through our brief stint with using real hardware components to support vibration capabilities, we gained exposure to basic components including breadboards, ESP32s, and vibration motors.
What's next for ARCane
In the future, we would like to expand on the open-source map sharing feature by allowing the user to automatically access shared maps by detecting their location and determining if another user has already mapped out that location.
Log in or sign up for Devpost to join the conversation.