Nobody likes to look at messy handwritten math, whether that be the TA grading your homework, your friends who missed a lecture, or even yourself when you're trying to study. Whatever the case be, a properly formatted and typesetted document works better for everyone.
What it does
isomath is a tool for converting handwritten math into Latex documents.
The tools is simple and easy to use: Take a picture of the math, upload it to our tool, and download the resulting Latex document. We also spellcheck your document, making sure that all your spelling mistakes are fixed.
How we built it
In the backend, our tool relies on OpenCV to extract blocks of math equations and text from uploaded documents, then calls the Mathpix API to get the equivalent Latex. Afterwards, we assemble the various blocks of Latex into a single Latex document, we use Azure's Bing Spell Check API to correct any typos before feeding it into Latex to generate a pdf. This is sent to a pipeline that uploads the generated PDF to Azure blob storage and creates a cropped preview image of the PDF that is also stored in Azure.
The front-end was created using React and was built to be responsive and functional on mobile.
Challenges we ran into
Since we had little experience with OpenCV, it was challenging to properly extract math equations line by line, so that Mathpix could properly identify the math in each line. We initially wanted to create a mobile app with React Native but settled with a responsive web-app that works well on both web and mobile.
Accomplishments that we're proud of
- OpenCV works!
- Clean UX design
- App can be used flexibly and reliably
- Complex back-end pipeline
What we learned
- How to use OpenCV to extract blocks of features
- How to configure and use the many different offerings from Azure (Blob Storage, VM, DNS, Bing Spell Check)
What's next for Isomathic
- Increasing the efficiency of the processing pipeline
- Improving accuracy of equation recognition
- Translate document into desired language
- Verifying the equity of math equations in the image
- Find URLs to websites related to concepts in the document.