Close to 300 million people on earth live with a visual impairment, among which 39 million are blind. In Switzerland only, 320'000 people are visually impaired. When evolving in new environments, blind people struggle to understand the surrounding objects and avoid obstacles. For this reason, we have built SoundMap.
What is does?
SoundMap is a smart wearable belt for blind and visually impaired, equipped with a camera, that provides real-time information on the surrounding environment through Audio Augmented Reality. Similarly to the way we easily identify the position of whistling birds, we aim to scan in real-time the surrounding environment and produce directional sounds (through audio AR) in the earphones connected to the device. Blind and visually impaired people are, therefore, able to map and understand their environment.
How we built it
Prior to ICC, we met with the SBV-FSA, the Swiss federation for blind and visually impaired people. We assumed that we knew nothing about the expected features of an assistive device for blind and visually impaired people, and came up, after this first meeting, with a sketch of what the device should look like. We also ordered the required hardware.
During ICC, we assembled the hardware, designed and 3d-printed a case for the device, programmed the object recognition, the scene segmentation, and the directional sound features. We then assembled everything and obtained a working prototype. This was now time for us to test the device. After several software adaptations, we were able to locate main objects and people while closing our eyes. We were able to travel through the building and avoid collisions. But we also needed to test this in real conditions. So we went to Bern and presented the project to Luciano, the head of Innovation of the SBV-FSA. The overall feedback was really encouraging, Luciano could locate people and chairs easily in the room. We defined the next steps for the project, and we are currently integrating the required hardware for these steps.
Challenges we ran into
As soon as a project involved hardware, things tend to be more complicated. We struggled to integrate the audio feedback in the Raspberry Pi, and had to deal with limited computational power. Therefore, we integrated an additional TPU to accelerate processing and reach close-to-real-time processing of an image.
Finding the right hyperparameters of our models to have it working best in practice is also tricky but after several tests, we were able to identify the correct values.
Accomplishments that we are proud of
Meeting Luciano, a blind person, and having him to test the product was great. He could point the direction in which people and chairs were standing after just a few seconds testing the device. This was a strong validation of the hypothesis we made and the work we made until there. We also obtained a meeting with the Ophthalmic Hospital Jules-Gonin in September after a long discussion on our technology, and they are willing to test the device.
What we learned
We tried to apply as much as possible a customer-centric development of the device. Meeting with end-users from day one was probably the most valuable move for us. We also learned that hardware tends to make things more complicated, even though we tried to keep it as simple as possible. We have changed 3 times the way to wear the device, and ended up deciding that the device could be worn as the end-user wanted to, since there are different ways to walk with a white cane, and that sometimes, it could block the camera of the belt.
Finally, we learned more about the daily struggles and the life of blind people, which is probably the most important take-away for the years to come.
What's next for SoundMap
SoundMap is joining the Science2Market program from EPFL Innovation Park in September. We will be adding hardware to the device for better obstacle detection, ordering more units, and testing the product with SBV-FSA and Hospital Jules-Gonin. We will fine-tune our algorithms based on these feedbacks, and we plan to reach a fully-working prototype in 2021.