The TikZ package for LaTeX makes it possible to create beautiful, high-quality, professional looking graphs and other diagrams. However, anyone who has tried to create a nontrivial graph in TikZ knows that it's extremely tedious to manually place vertices, and has struggled with TikZ's opaque positioning rules.
What it does
Given an image of a hand-drawn graph (say on a piece of paper, or on a blackboard), Paper2LaTeX finds the locations of graph nodes and text elements in the image. Then Paper2LaTeX detects semantic information about the edges in the graph. Once it has this information, it generates the LaTeX code to display a high-quality version of the original input graph.
How we built it
We used OpenCV for the initial image preprocessing, and for detecting nodes in the graph. Once we have that, a novel search algorithm that we developed detects semantic links between nodes. We also trained a custom convolutional neural network to recognize handwritten letters and digits, to create text labels.
Challenges we ran into
One of the main challenges that we had to solve was dealing with crossing edges in the input graph (not every graph is planar!). This required rethinking and modifying our edge detection algorithm to support curved edges, while still finding the correct path when two edges intersect.
What's next for Paper2LaTeX
In the future, we hope to improve the interface for Paper2LaTeX to make it as seamless as possible, as well as supporting additional features such as different shapes for nodes.