Inspiration
We often use soundbites to communicate with our friends on voicecalls, and we were wondering about how we could replicate this experience in the real world. We thought that this could be an interesting way to enhance human interactions. It could be a tool for entertainment purposes, producing sound bytes for comedic effect. It can also be used to enhance non-verbal communication. In the future, for example, this technology could be used to translate sign language in real-time.
What it does
The Seeing Soundboard uses an image-based gesture classification algorithm to detect different hand gestures, such as a thumbs up or a thumbs down. The images are collected via the camera module of a Raspberry Pi. With the push of a button, the device is connected to its power source, and the rasberry pi runs the algorithm and dishes out a sound to a speaker module.
How we built it
We set up the RasPi and tried to connect it to a camera module. We connected the speaker module to an esp32 microcontroller, and we have the esp32 connected to the RasPi After that we connected the speakers and 3d printed the housing for the device which is connected to a lanyard so that it can be carried around a person's neck.
Challenges we ran into
One of the most time consuming things on our list would have to be setting up the Raspi since hardware almost never works correctly the first time. We also tried testing out the speaker module using an esp32 in Arduino IDE, and we couldn't figure out how to boot the microcontroller into the IDE for the longest time until one of us simply just, figured it out. Intergrading Viam into our workflow was also quite challenging.
Accomplishments that we're proud of
There is real potencial to be found with the use cases of this device. It would be very cool to think that there is a potencial world where mute or deaf people can easily communicate with people who don't know sign language.
Also, one of our team members have stated that they are "no longer scared of RaspberryPi. Just pissed off."
What we learned
We have had the opportunity to get familiar with the larger-scale components of the RasberryPi.
In addition, a certain team member has expressed that they learned the following: "sometimes it's a hardware issue, and you can't do anything about it."
What's next for The Seeing Soundboard
More comprehensive sign language is likely the logical next step in developent (assuming we finish building the thing).
Log in or sign up for Devpost to join the conversation.