vocal space explorer

Inspiration

We had two main inspiration, we wanted to create a program that is able to train our voices and we wanted to create a program that allows people with disability a way to play video games. The idea was to create a control system that uses our voices to replace mouse/keyboard. This allows us to improve our vocal control through long-term use of our control system as well as allow people with physical disabilities an alternative way of interacting with computers.

What it does

We created an engine that takes a live audio stream and converts it into acoustic features (pitch and formant frequencies). These features are then used as control inputs to various game demos.

How we built it

The game was built in python. We first made a prototype to record a live audio stream from the computer's microphone. With our knowledge of signal processing and human speech, we created an algorithm that calculates acoustic features from audio samples. Once this was complete we tried to create games and replace the traditional mouse/keyboard control scheme with the output of the acoustic engine.

Challenges we ran into

The software has difficulties in noisy environments so a lot of time was spent to improve the accuracy of the acoustic measurements. Integrating our acoustic engine to replace the traditional mouse and keyboard controller was not as intuitive as it seemed since by default most input type involved the mouse and keyboard only. Collaboration difficulties to ensure everyone was on the right page.

Accomplishments that we're proud of

Creating an acoustic engine that is fairly accurate in a noisy setting. Ability to interface voice as inputs to replace mouse and keyboard

What we learned

We learned how to create a pygame game. How to interface with a microphone. clustering algorithms for categorizing voice data. interfacing custom input types for controls in pygame.

What's next for vocal space explorer

We want to further improve our system's accuracy. Add additional acoustic features for control such as phoneme identification, intensity of voice, tonation, etc. Create more sophisticated games such as accent training.