WeSee
A portal hardware to aid visually impaired people to see the world by getting photos in front of them and translate those visual images to voice messages
Inspiration
We are all seeing the world in one way or another. Some use vision, some use hearing and some use touch. We use our eyes to see because we can acquire visual images. But those who are visually impaired need to depend on other senses to translate into images in their brians. Our goal is to use technology to aid people who have vision problems to acquire more information from the outside world via voice messages.
What it does
WeSee translate images to voice message and let user to “see” and “read” things around them. One highlight of WeSee is detecting the object and its distance. It tells the user what is the object, its distance and direction ahead of him. The object is ranked by its danger level and the user will receive the voice information of the most dangerous object first, which helps the user to avoid the potential danger. In addition, WeSee distinguish objects from human. If it is a person, it can detect his facial expressions and emotions, which greatly aids the user to interact with people.
How it does
WeSee uses a webcam to take real-time photos and update them to Google Vision API to detect the objects and texts within the images, and combines with the distance information transmitted from the ultrasonic sensor to generate text messages locally. We built an algorithm to rank the danger level of an object and prioritize to deal with the most dangerous object. Then WeSee sends this message to Google Text to Speech API to process and then sends back to local and outputs as audio to the user’s earphones.
Challenges we ran into
- Change from Java to Python because Raspberry Pi is ARM and did not support our protocol
- Connect the ultrasonic sensor to the Raspberry Pi
- Failed to output the audio from local
- Deal with the delay time from the Google API
Accomplishments that we are proud of
We are proud that we managed to finish both the hardware and software part of WeSee within 24 hours. The most accomplished part is that we managed to categorize and select the text messages we want from API and output as an audio. We enjoyed the time to work and solve problems with a group of efficient and intelligent people and we are glad our product can help visually impaired people.
What we learned
We learned a lot about Google APIs, Raspberry Pi, changing environment from Python to Java and connecting a sensor. We also learned to work with a team of people never met before, and it was the first time for some teammates to attend a Hackathon. From brainstorming, distributing work and solving problems together, we learned how to communicate and collaborate.
What is next for WeSee
- More advanced camera setting(sport settings, night mode and etc) and more portable and convenient design
- Faster speed for transmitting and interpret data
- More advanced text analysis
Log in or sign up for Devpost to join the conversation.