What is Jackbot?

Jackbot is short hand for programmers, allowing them to create a basic outline of what they want to do. We highly recommend that you read our docs as it gives all of the available commands. Documentation

How we built it

We built this piece of technology using mainly Javascript, implementing parsing sequences and data structures for holding the generated DOM. We used Webkit's speech recognition callbacks, which are currently built into many modern browsers, to listen to speech and react according the user's input. Our system defined a set of guidelines for the user to follow when "speaking" code, which greatly reduced the ambiguity of some statements when translated to code. We also update a rendering of the code in real time using an IFrame.

Challenges we ran into

During the development process, we encountered a couple challenges in generating JavaScript and CSS from speech. HTML was easier, because it already has a formatted style, but JavaScript and CSS were harder because there were a lot more ways for users to organize their code.

Accomplishments that we're proud of

One thing that we are proud of about our project is it's ability to recognize and replace common mishearings. We created Jackbot with this in mind, knowing that the basic browser speech recognition software is not extremely advanced. We are also proud of Jackbot's ability to render the code that you are creating in real time. This allows for a better user experience, and makes the webpage almost a standalone editor.

What we learned

One of the biggest hurdles that improved our coding skills was the implementation of structured data to represent the HTML, Javascript, and CSS. To be able to edit the code blocks with efficiency, we created a data structure to hold each element of the generated code. This improved efficiency by eliminating the need for parsing the HTML every time we wanted to edit it. It also provided opportunities for highlighting the current element/line of code, which improved the overall user experience as well.

What's next for Jackbot: Speech to Code

We definitely want to work on making the voice recognition better through neural nets. Along with that, it be great to add other programming languages so that we can appeal to a bigger audience.

Share this project: