Today, computer programmers all over the world have one of the highest rates of Carpal Tunnel Syndrome. Studies have shown that greater than 30% of adults experience sensory symptoms due to typing on a regular basis. This has necessitated many programmers to spend large portions of time attempting to build the optimal ergonomic setup, potentially involving adjusting chair height, ensuring your back is straight, purchasing ergonomic keyboards etc. However, the problem can still exist, and remain a significant hindrance in ones productivity. To alleviate this, we proposed the idea of removing the need for a keyboard altogether, through the use of speech to code conversions. We hope that by providing a robust and easy to use speech analyzer, programmers will no longer need to constantly use a keyboard and undergo the unfortunate consequences.

What it does

Ava provides many features related to typing within an editor. The main feature is the ability to directly add code to an open file by simply saying the code they want to include. Another important feature is the ability to make aliases, for example psvm (Public Static Void Main), enabling use of aliases rather than saying all the words. Additional features include cursor movements, running terminal commands, and commenting/deleting specific lines. Ava can also tell you cheesy programming jokes!

How we built it

We built a VS Code extension using JavaScript. The initial step of speech to text conversions is done by using the Google Cloud Speech to Text API. The text is then processed by our custom built natural language processing engine to understand the context and perform said commands.

Challenges we ran into

We had initial challenges setting up the Google Cloud Speech to Text API. We had to work with not-so-good laptop microphones in a noisy environment, which often caused the API to misunderstand speech.

Accomplishments that we're proud of

Our project involved many complicated aspects, mainly in parsing and understanding the converted text and determining the commands to run. Consequently, it was very gratifying when we saw all the features working together and as we had envisioned.

What we learned

We learned new aspects of building VS Code extensions, creating our own natural language processing engine, and utilizing Google Cloud API.

What's next for Ava (Amazing Virtual Assistant)

We hope to add support for more language specific features to Ava and support a larger array of editor commands. We also want Ava to have the ability to perform voice searches for programming questions and show and integrate search results into the editor.

Built With

Share this project: