Jamie teaches a section of Intro to Computer Science at Rutgers and she asks a lot of her students to handwrite code in preparation for exams. She found it tedious to have the students hand write their code then transfer it over to their computers, so she decided that this would be a great use for students and also those preparing for interviews!

What it does

Our mobile app takes either a picture or a picture from the gallery of hand-written python code and converts it into runnable python code via Microsoft Azure, compiles it, and displays the results to your phone.

How we built it

We split this task into 4 phases. 1) Get a picture of code and color contrast it to black and white (remove the shadows) 2) Pre-process the image to split the code line-by-line and account for accurate tabulation via flask. Pass each line of code into Microsoct Azure for OCR handwriting-to-text. Pass this code back to mobile to verify the converted code. 3) Pass a file via HTTP GET and executes as bash command to compiles and runs the code converted from the image. Uploads a gist to github to be accessible at a later time. ("this is a node.js server that does everything" - Carlin) 4) Display results on mobile

Challenges we ran into

  • Qasim couldn't figure out a weird bug that auto-rotated a picture, but he figured it out!
  • A lot of the OCRs we tested were directed towards font-styled images, so converting handwritten text to editable digital text was a bit challenging. Therefore, we included an opportunity to edit/correct the code in the mobile app before it's compiled and run.
  • OCRs have difficult times recognizing special symbols, like =, {, }, (, ), etc. that are vital to coding, so some of them may not appear in our converted text

Accomplishments that we're proud of

  • Qasim made an image contrasting via his mobile app
  • David and Jamie worked on our own logic to separate each line of code and incorporate indentation for our python code

What we learned

OCRs are currently not powerful enough to accurately convert handwritten text to editable digital text, so it was a little difficult to find an OCR that was accurate enough. David and Jamie learned how to process an image into little pixels and develop various methods to separate lines without hard-coding in the pixel count because hand writing varies a lot.

What's next for Descartes

We currently only support python because of the difficulties for recognizing hand-written curly braces and other special symbols. We hope to support more languages in the future and implement a better system for recognizing different symbols.

Share this project: