VoiceCraft

voice craft caption

Inspiration

Our inspiration came from an interest in working with video games, such as mastering a game or creating a new approach to a classic video game. We decided to go with Minecraft since it is one of the most iconic games of all time. This combined with its simplicity made it an ideal choice, with a realistic yet challenging goal. The Assembly AI voice recognition software was the most interesting of the challenges available and we wanted to come up with a unique method to utilize that in a way that has not been done before. While popular devices such as Google Home or Amazon Echo that interface with the real world, we decided to create an application that can control your computer with just your voice

What it does

Our application Voice Craft allows you to use voice commands that are connected to key binds or mouse movements. This allows you to play a game such as Minecraft since all it requires are simple key binds and several mouse movements. In fact this program can be configured to work for many different types of games and even can be used for general computer use.

How we built it

The transcript generated by the AssemblyAI API is analyzed for keywords corresponding with the user commands. Each of these commands are linked to keyboard and mouse events allowing each voice command to be virtually instantaneously converted into an input recognized by the computer and registered in a game or app. We built the graphical user interface part of the application using a library called pysimplegui would not recommend. Adding and deleting commands, keys and movement were done by connecting a table to jscon file and updating the table afterwards. Choosing an audio device was done by receiving a list of input devices from pyaudio and then selected them by there index.

What's next for VoiceCraft

Adding longer, custom phrases would allow for a wide range of commands. Further, increasing the consistency of the audio detection and decreasing the input delay would allow for better responsiveness overall. Once those have been improved, the possibilities would be limitless, as the current customizability was maximized for future improvement.