Inspiration

We've noticed that many educators draw common structures on boards, just to erase them and redraw them in common ways to portray something. Imagine your CS teacher drawing an array to show you how bubble sort works, and erasing elements for every swap. This learning experience can be optimized with AI.

What It Does

Our software recognizes digits drawn and digitizes the information. If you draw a list of numbers, it'll recognize it as an array and let you visualize bubble sort automatically. If you draw a pair of axes, it'll recognize this and let you write an equation that it will automatically graph.

The voice assisted list operator allows one to execute the most commonly used list operation, "append" through voice alone. A typical use case would be a professor free to roam around the classroom and incorporate a more intimate learning experience, since edits need no longer be made by hand.

How We Built It

The digits are recognized using a neural network trained on the MNIST hand written digits data set. Our code scans the canvas to find digits written in one continuous stroke, puts bounding boxes on them and cuts them out, shrinks them to run through the neural network, and outputs the digit and location info to the results canvas.

For the voice driven list operator, the backend server's written in Node.js/Express.js. It accepts voice commands through Bixby and sends them to Almond, which stores and updates the list in a remote server, and also in the web user interface.

Challenges We Ran Into

  • The canvas was difficult to work with using JavaScript
  • It is unbelievably hard to test voice-driven applications amidst a room full of noisy hackers haha

Accomplishments that We're Proud Of

  • Our software can accurately recognize digits and digitize the info!

What We Learned

  • Almond's, like, really cool
  • Speech recognition has a long way to go, but is also quite impressive in its current form.

What's Next for Super Smart Board

  • Recognizing trees and visualizing search algorithms
  • Recognizing structures commonly found in humanities classes and implementing operations for them
  • Leveraging Almond's unique capabilities to facilitate operations like inserting at a specific index and expanding uses to data structures besides lists
  • More robust error handling, in case the voice command is misinterpreted (as it often is)
  • Generating code to represent the changes made alongside the visual data structure representation
Share this project:

Updates