Parseltongue

Inspiration

Parseltongue was inspired by the desire to encourage computer science education in frequently marginalized groups. Whether they possess physical disabilities such as struggling to read, type, or focus, Parseltongue aims to alleviate any setbacks they might face when learning how to code.

For those who have read the Harry Potter, Parseltongue is the language of snakes. Since our application’s code is in Python, we thought our web app would be aptly named ParselTongue. Moreover, we hope that the Harry Potter-inspired theme provides a point of connection with users of a younger demographic who are perhaps inexperienced and feel daunted at the prospect of coding; we hope to simplify and entertain their learning process as they discover that coding can be fun, even magical, and yet easy to understand.

What it does

Parseltongue is a speech-to-code web application that allows the user's device which utilizes its microphone to record speech. Consequently, Amazon’s Transcribe converts speech to text, then our JavaScript code converts the text into Python code. The user can then download the Python file and run it on their local machine. By using speech as a means to code, we strive to help those with physical or mental disabilities that makes it difficult to write code through the traditional means of typing.

How we built it

We used Amazon Web Services to host our website, and Amazon Transcribe to convert speech to text. We deployed to AWS Amplify by connecting our git repository to the service, and routed it through Route53 to use the domain we got from Domain.com. Ace Editor was embedded onto our website as a text editor. The backend of our application is written in JavaScript, and HTML and CSS were used to create and style our website.

Challenges we ran into

The biggest challenge we had was deciding if our application should be created with convenience or education in mind. For convenience, users would be able to say voice command shortcuts that would automatically insert parentheses, brackets, and commas when needed, or write simple function calls when given a command. On the other hand, for education, users would be forced to speak out every character they wish to insert, such as saying “print open parentheses x close parentheses.” The purpose of this is because syntax is extremely important when coding, and users must internalize how to properly write code. In the end, we chose to focus on the educational aspect because in this day and age, syntax is essential. However, to help users who may be inexperienced with python, we added an instruction book at the end with several pages that elaborate on basic, important concepts such as print statements and if/else statements.

In terms of technical challenges, we had difficulty implementing AWS’s Transcribe API. To overcome this challenge, we followed Amazon’s example application which used browserify to build the application. However, this choice caused us to be unable to run the final Python file in our server.

Accomplishments that we're proud of

For this hackathon, we’re extremely proud of learning and implementing a new technology. Using Amazon Web Services was an enriching experience as we were able to implement their software and learn even more about their innovations in the industry. We are also happy that we were able to gain experience in different languages, such as JavaScript, which we seldom use in our current course curriculum.

What we learned

We learned a lot about the limitations of speech recognition technology, and just how difficult it is for computers to mimic humans in their listening capabilities. Speaking one thing and seeing another being displayed on the screen was at times frustrating.

What's next for Parseltongue

Now that we have implemented the educational side of Parseltongue, we are also very interested in creating the convenience mode for more experienced coders or those who are extremely burdened by the syntactical nuances.

After polishing our technology, we hope to encourage a wider audience to use Parseltongue for educational or practical purposes.