We were inspired by Scratch as a way to make coding more accessible to beginners, and we wanted to make something to help get people started with functional programming.
What it does
Itch is a programming language where you drag to connect inputs and outputs to make a program. Since Itch is a functional language, it lends itself intuitively to being represented as a graph, which makes it very clear how information is flowing through the program. With its intensely visual approach, Itch is a great tool to learn about functional programming, which can often seem quite abstract. Furthermore, Itch's advanced voice controls allow you to program without touching the keyboard, a great pro for those with disabilities, programmers who are tired of typing all the time, and people who are looking for a fresh, fun way to program!
How we built it
Itch's frontend was built with Vue in Electron using a library called elevtron-vue. On top of that, the main user interface is a heavily modified version of a graphing library called rete.js. Among the things we added were support for multiple, resizable workspaces, expansion of nodes, and lazy evaluation of the user's code. Speech transcription was done with the Google Cloud Speech API, and Stanford CoreNLP was used to grammatically categorize words and phrases in the text. By picking up on parts of speech, interpreting numerical values, and identifying phrase and word dependencies, we were able to put together our own algorithm to transform spoken text into a series of instructions to manipulate the on-screen graph.
Challenges we ran into
Formalizing the commands we needed to support to ensure an intuitive user experience without bloating our software too much was a fairly large hurdle. On top of that, developing a linguistic method to analyze speech and turn it into commands was much more challenging, and probably the hardest part of developing Itch. While we initially envisioned a machine learning algorithm for this application, we then realized that a more deterministic approach would achieve higher consistency for the type and number of commands we wanted to use; the commands given to the system and its output were relatively structured. By honing in on the sentence structure of the input text, we were then able to allow the user to execute multiple commands in one phrase and ensure a reasonable interaction with the graph. Another challenge was the sheer number of frameworks we had to learn. Most of us had never seen any of the libraries we used here before, and we ran into many technical difficulties getting everything set up before we really got to work. We got around this with some careful division of labor, but ran into even more problems integrating combining our work, with no knowledge of the other pieces. However, we were able to agree upon a few standard formats that made putting everything together much easier, such as our command encoding.
Accomplishments that we're proud of
We're very proud that we got this project working at all. It was pretty ambitious jumping in head first to all of these technologies, but we're glad we ended up with a fairly polished project in the end. We're also proud of switching away from machine learning, even though we really wanted to do it, because our new solution is much more suited for our application. Finally, we are very proud of pulling through for this project. We got off to a rough start on Friday, but we've been working our butts off over the past 24 hours, and we think it paid off.
What we learned
Besides learning how to use all the technology here, most for the first time, we've all learned how much we can get done in 24 hours if we really sit down and work. We also learned how to build something better as a group by using each others' strengths and experience to the fullest.
What's next for Itch
We would love to add to our voice control system and incorporate even more interesting, compound actions to make coding in Itch even more fun. Additionally, we think being able to import functions other people have made with your voice would make this tool even cooler.