We hope to design an app that works as the "eyes" of the visually impaired people and help them locate and reach objects without the help of others.
We spent almost 1/4 of our time installing OpenCV on Raspberry pi. The whole process is unbelievably toxic.
How we built it & How does it work
Our app helps the visually impaired to find objects in real world. Users can tell our app which object they want to find by vocal, and our app will give out voice instructions in turn. The user will need to equip a camera and the raspberry pi. Our app is pre-trained with database of commonly seen objects such as table, car, dog, stc., and it would also be easy for the user to select an object that is not present in our supported trained data and train them directly, in future.
Use Cycle: The user's command will be processed by the Azure Speech-to-text REST API and then propagated to raspberry pi with Socket. Then our raspberry pi will process the commands from camera and radar information with OpenCV and Deep Neural Network, and then propagate data back to the cellphone using Socket as well. Finally, the phone will read out the command using Android Text-to-Speech API. We use Android (Java) for user end and Python for the raspberry pi.
Press the cellphone screen and say finding instructions containing the keyword "find", and the cellphone will confirm with you and then provide you with instructions about how where the object is. It will then enter into the "ranging" mode and direct you to the object through manipulating the frequency of beeping.
Our app is designed based on the needs of the visually impaired people. It does not require any eye perception to operate. The only operations the user need to do is pressing the phone and speaking. That's all.
We hope to improve our DNN model and add more to our trained database. We are also looking forward to completing the custom training process to enable instructions for almost any object.