We thought it would be helpful for students to be able to take pictures of their notes from math class or their professor's chalkboard to get a nice formatted Latex version of their notes. We also thought there were some very interesting challenges involved including defining keywords to parse from handwritten words to our custom syntax.

What it does

You take a picture of your math notes using certain reserved keywords that you define as preferences as a user of our website. Once you upload this picture to our website, we will spit out a formatted latex document that maps notes to a formatted latex document using the syntax that you set as a user of our website.

How we built it

We used the Google Cloud Vision Api to identify handwritten words from an image. We used Kmeans clustering in order to group and order the words in the order as they were meant to appear on the paper (the algorithm was provided by SkLearn). We created a website for users to login, upload their photos of their math notes, and store their preference maps, using Node.js, MongoDB, Handlebars, and Express. Once an image was uploaded and the words and their ordering were identified, this information was sent over to a Python file which parsed and 'compiled' the handwritten syntax (using special reserved keywords that the user defined in a preference map) down to Latex. For instance, the user may want to write 'nl' in his/her notes to denote the start of a numbered list. When we are compiling the handwritten code to Latex code, we parse this 'nl' token and generate a Latex token of the form '\begin{enumerate}'. Once the handwritten code is parsed and compiled to Latex, the viewable Latex file is written to the current directory for viewing and further use.

Challenges we ran into

We ran into several challenges getting Google's Cloud Vision Api to play nicely with our custom defined syntax, figuring out how to group words by order from an image, parsing the handwritten code to LaTex, and making the whole process smooth for the user. It was also hard to figure out how to let the user define their own syntax which wouldnt be intrusive to their normal note taking styles. This was a really interesting issue, but we found certain keywords which were short, concise, and not used at all ('nl', 'eb', ...).

Accomplishments that we're proud of

We implemented some interesting algorithms to preserve word order when retrieving words from an image file, so we definitely are pretty proud of that. We are also really proud of the parser we wrote which takes a preference map from a user, user-defined syntax retrieved from the image the user uploaded and generates the working LaTex code. It's really neat to see it all come together!

What we learned

We learned how to use the Google Cloud Vision Api, how to use kmeans clustering to solve a real problem that we were encountering, how to preserve word ordering when retrieving words from an image, and how to work on a new problem that neither of us have any experience with.

What's next for TexToLatex

We want to extend it's parsing and compilation capabilities to support the full latex syntax and hopefully external packages. There are a lot of exciting features we are thinking of adding!

Built With

Share this project: