Project Title: FISH
Finding Items & Spaces Harmoniously
Inspiration
Our inspiration for this project came from observing how fish navigate in their underwater environments. Unlike humans who identify markers to determine their position in the area, fishes navigate by identifying the objects around them.
Just as fish rely on their senses to navigate the vast ocean, we aim to empower individuals with low vision or blindness to navigate their surroundings with confidence.
After choosing to target Vision as the pillar of Accessibility we wanted to focus on, through various conversations with friends in these communities and research, we found that the majority of these individuals were of low vision, and a small handful were actually completely blind. This meant they had various medical conditions from tunnel vision, to cataracts, that affected the way they perceive the world.
To better understand how these users interact with their devices, we decided to try VoiceOver, the built-in Screen Reader on iPhone as well as typing in braille. This gave us a better sense of how those with low vision or blindness interacted with their devices through a series of gestures with existing technologies.
This background research, coupled with the extensive discussions with people in the low vision community and accessibility advocates from various major technology companies helped us gain a better understanding of the problems that they face day-to-day and how we can build a solution more tailored towards their needs.
Be My Eyes: Similar to the Be My Eyes app, we can integrate a live support system with our solution to provide real-time assistance where a support agent would be able to view what the user is seeing and guide them through, as relying entirely on machine learning can be inaccurate at times.
What it Does
FISH (Finding Items & Spaces Harmoniously) is a cutting-edge assistive technology designed to provide audio descriptions of the user's immediate environment.
By summarising the surroundings and emitting text-to-speech audio, FISH helps visually impaired individuals gain a better understanding of their environment, enhancing their independence and safety.
How We Built It
We've leveraged a combination of technologies to create FISH:
- Gestures & Video: We utilised iPhone Swift to capture video input and interpret gestures, making it user-friendly and accessible.
- Object Detection: We integrated YoloV8X, a state-of-the-art computer vision model with python for real-time object recognition.
Challenges We Ran Into
During the development process, we encountered several challenges, including:
- Integration Hell: Working on a prototype across various disciplines from iOS development, machine learning, cross platform interaction etc., integrating them to create a single product that feels magical to a user was a challenge.
Accomplishments We're Proud Of
We're thrilled to highlight these accomplishments:
- Object & Gesture Detection: We successfully implemented object and gesture detection, enabling FISH to provide rich and accurate descriptions.
- Optical Character Recognition**: Non-intrusive detection of objects in surroundings with text integrated with speech-to-text
- Hardware & Software Integration: Seamlessly combining hardware and software components for a cohesive user experience.
What We Learned
Throughout the development of FISH, we gained valuable insights and skills, such as:
Swift Development: We deepened our expertise in Swift, enabling us to create a robust iOS application for gesture detections, and providing a high-quality video stream for object detection.
YoloV8 Model Integration: We learned how to harness the power of YoloV8(X) models for a fast abd responsive real-time image and object recognition.
Cross-Platform Communication: We mastered the art of handling communication between our Swift application and Python application, providing a seamless user-experience.
What's Next for FISH
The journey doesn't end here at WTH. We have ambitious plans to enhance FISH even further:
Implementing Lidar: We aim to incorporate Lidar technology to provide depth distance estimation, offering a more detailed understanding of the surroundings.
Improving Object Detection: We plan to refine the object detection algorithms by improving the model's dataset to have a wider range of objects that our software would be able to detect. (Road Signs Detection, etc.)
Potential Additions: Sign Language Translation: We envision enabling FISH to translate sign language, facilitating communication between visually impaired individuals and those with hearing impairments.
Potential Additions: Braille Translation: Another exciting possibility is implementing Braille translation, allowing visually impaired users to read Braille without physical contact.
FISH is more than a hackathon project; It's a promising step towards improving the lives of visually impaired individuals. We are excited about its future and the positive impact it can have on the world.
Log in or sign up for Devpost to join the conversation.