There are a lot of things we do in our lives that rely on our eyes, especially as programmers.
It came as a surprise to see a blind student in our classes. As much as companies and schools try to help people with disabilities, they have limited staff and resources.
When we learned about the City of Mississauga's Smart Cities initiative at PC Hacks, we thought of a way to make the project be more accessible and inclusive.
What it does
We have developed an assistive technology application to help the visually impaired through live remote assistance from their mobile device called BlindSight.
Volunteers can help the visually impaired enhance their daily lifestyle by allowing them to be more independent. Routine tasks like reading medicine bottle labels, navigating commute to and from work are more difficult to accomplish with limited vision.
Core Features - object recognition, live video chat, gesture navigation, voice assistance, torch toggle
How we built it
Swift 4 - iOS app
Core ML - object recognition
Twilio - video interface
Google Maps API - location recognition
Firebase - push notifications, server, database
Balsamiq - Wireframing for UI/UX
Challenges we ran into
Machine Learning - Accuracy of objects recognized in the camera's view
Server Connection - Firebase
What we learned
UI/UX Design - keeping UI simple to make a blind person's UX as smooth as possible (limiting buttons in case of errors - as the usual best practices don't apply to people with disabilities; privacy (disabling front-facing camera)
What's next for BlindSight
Android App - to increase user base
Reward System - Partnering with sponsors to provide rewards for reaching helper milestones (number of hours, number of users helped, maintaining a high rating), and schools for community volunteering recognition
Emotion Detection - Recognizing user's emotion via facial recognition for immediate user feedback
Voice Commands - User-friendly navigation app
Location Pairing - Allowing volunteer to select specific user to help from a map