SpeakCode

SpeakCode is an online JavaScript IDE that aims to make programming more accessible for people with disabilities in their hands and/or arms through the power of speech recognition.

Comment

Inspiration

By 2020, it is estimated that there will be 1 million more computer science jobs than students. Unfortunately, there are over 1 million people in the USA with a disability in their hands and/or forearms. Our team recognized a need to bring tools to market that promote a computer science scene that is more inclusive for our friends with disabilities.

What it does

SpeakCode is an online IDE that allows the user to both program and run JavaScript programs using their voice. It supports parsing speech intended to produce code for major programming concepts, including variables, loops, and conditional statements. Common edit commands are also available via voice activation, including navigation, deletion, and insertion. Even auto-indentation is supported! To see the output of a user's code, all they have to do is say the word, "run." (Typing and mouse control are supported as well - we hope to make this platform accessible to as many people as possible!)

How we built it

SpeakCode is a website hosted on AWS Elastic Beanstalk. It is built using Node.js and the Express framework for server-side operations, as well as the Bootstrap framework for the front end. With the user's consent, the IDE receives a constant stream of data from the user's microphone to feed into the Microsoft Azure API for speech recognition. The text transcription is then parsed using an algorithm we developed to identify, structure and display different programming features in the IDE.

Challenges we ran into

This was the first time any of us had ever created a web application, and so we had some initial trouble understanding which technologies to use. We especially ran into trouble attempting to incorporate a Web Socket/networking aspect, i.e. how we would communicate between the front end and back end. After a while of attempting to pursue a heavy server-side solution, we realized the project could be accomplished much more simply and with much less latency using local JavaScript. We were very grateful for the assistance we received from all the mentors here at SD Hacks who helped us to make such key decisions regarding our project.

Accomplishments that we're proud of

Here is what each of our members are proud of about our project. Dawson: I never thought I would ever see keyboard-less programming and yet here we are developing it ourselves. The concept behind this project is simply the most astonishing idea for me. Sachinda: The fact that we created a tool that does not really exist yet is really surprising to me. Jason: I'm extremely proud of our ability to bring our idea from idea to reality, as well as for our ability to collaborate with each other and reach out for help when needed

What we learned

All of us learned a ton about web development and how to create full web applications. Here are what each of us individually learned from the project. Sachinda: I learned a lot about building a full web app, as I had no prior experience. From the front end technologies such as HTML to back end tools such as Elastic Beanstalk, I struggled, but also became much more knowledgable about them. Jason: I learned what AWS, Node.js and Express were used for, and how to integrate these technologies with front-end technologies I was previously familiar with. Dawson: I got to experience firsthand what is like develop a complex natural language processing heuristic alongside gaining Node.js and AWS familiarity to complement my existing skills.

What's next for SpeakCode

There is major room for improvement with SpeakCode's parsing algorithm in order to support more JavaScript features. Additionally, we may seek to extend SpeakCode's language capabilities beyond web programming languages and further into computer programming languages such as Java and C++. We may also consider further ways we can improve the ease of use for disabled users (perhaps user interface modifications or additional voice-activated commands), especially with the help of feedback directly from those users.

Share this project:

Updates