We were inspired by a similar app, called PhotoMath, which can quickly produce solutions to complex mathematical problems based on a handwritten picture. We wanted to see what it would take to implement our own version of the app.
What it does
This program takes a picture of an arithmetic expression, drawn in a program like Microsoft Paint. Then, by segmenting the digits, running each one through a convolutional neural network to classify them, and evaluating the result, the program can determine the value of the expression
How I built it
First we started by trying to construct a postfix expression evaluator, to evaluate strings of symbols. However, we later learned this was completely unnecessary, as Python has its own built-in string evaluator. We then trained our Convolutional Neural Net on handwriting samples we collected from other hackers, by first augmenting the data set to provide many more samples. Finally, after segmenting the images into individual symbols, we could evaluate the handwritten expression
Challenges I ran into
The postfix expression evaluator, before we learned we did not need it, was very difficult to construct, especially taking into account floating point numbers and negatives. Then, when training the neural net, we had numerous problems with our collected training set. Since generation and training was a long process, this resulted in a huge time loss for each small mistake. The final model has some difficultly with some symbols, especially the "subtraction" and "division" symbols
Accomplishments that I'm proud of
We were successfully able to train a neural network that can evaluate handwritten expressions within a degree of accuracy. In addition, as an intermediate step, we were able to develop a procedure for quickly generating large amounts of training data from a small collected sample, which could possibly have many other useful applications.
What I learned
We learned about how to implement and modify neural networks using the keras and tensorflow libraries. We also got to practice data manipulation and lexical analysis while constructing the project
What's next for Handwritten Expression Evaluator
Next, we plan to add support for floating point numbers, and add additional features such as an algebraic solver and more symbols and operations such as exponents.