Notes2Tex

Transform angled, glare-filled whiteboard photos and handwritten notes into clean LaTeX and generative TikZ vector graphics, injected directly into Overleaf in one click.

Comment

Inspiration

As a STEM student, every fast-paced lecture ends the same way: taking a frantic, tilted photo of the chalkboard or scribbling down messy proofs and diagrams. But transcribing those visual notes into publication-quality LaTeX for problem sets or research papers takes hours of tedious manual typing. Typing out Greek matrices, integral bounds, and attempting to redraw complex system diagrams or state machines by hand drains time that should be spent actually understanding the material.

I wanted a tool that bridges the gap between raw handwriting and academic publishing without requiring manual copy-pasting or file re-uploading.

What it does

Notes2TeX is an end-to-end computer vision and multimodal synthesis pipeline with an interactive web studio and a Chrome extension for Overleaf.

  • Computer Vision Normalization: Takes real-world whiteboard and notebook photos taken at awkward angles under fluorescent classroom lighting, straightens them onto an orthogonal plane, and strips out glare while enhancing faded marker ink.
  • Layout Segmentation: Automatically isolates and classifies regions into Text, Mathematical Expressions, and Structural Diagrams.
  • Multimodal LaTeX & TikZ Synthesis: Converts handwritten proofs into valid KaTeX syntax and transforms hand-drawn sketches into editable, standalone LaTeX TikZ vector graphics.
  • Interactive Studio: A neo-brutalist web interface where clicking any bounding box on the original photo highlights the corresponding compiled math section or TikZ vector graphic in real time.
  • 1-Click Overleaf Chrome Extension: When working inside Overleaf, opening the extension and uploading notes or a PDF automatically compiles the document and programmatically injects the compiled LaTeX directly into the active CodeMirror editor at the cursor position.

How I built it

As a solo developer, I built the entire system across four integrated layers.

Computer Vision Engine (OpenCV & Python)

  • Perspective Keystoning: Uses cv2.approxPolyDP to detect document and whiteboard boundary contours, calculates a homography matrix with cv2.getPerspectiveTransform, and performs a perspective warp to flatten angled shots.
  • Glare Neutralization: Converts images to LAB color space and applies Contrast Limited Adaptive Histogram Equalization (CLAHE) to the L-channel, suppressing hot reflections while boosting low-contrast marker strokes.
  • Morphological Layout Segmentation: Applies horizontal projection profiling and connected-component dilation to cluster and classify regions into text, equations, and visual diagrams.

Multimodal AI & Vector Generation (Gemini 3 Flash & Tesseract)

  • Uses Gemini 3 Flash (gemini-3-flash-preview) with structured prompt engineering to parse mathematical proofs into standardized LaTeX syntax.
  • Rather than converting diagrams into raster images, the model synthesizes standalone, vector-based LaTeX TikZ code for state machines, memory layouts, graphs, and circuit diagrams.
  • Features a resilient cascade fallback across gemini-3.5-flash, gemini-2.0-flash, and local offline Tesseract OCR.

Web Studio (React, Vite, Tailwind CSS)

  • Neo-brutalist grid-notebook user interface.
  • Interactive SVG canvas synchronizing detected bounding boxes with live KaTeX rendering and compiled TikZ previews.
  • In-memory multi-page PDF rasterization via pypdfium2 with a dedicated slide pagination toolbar.
  • Built-in raw .tex source editor with syntax line numbers and 1-click clipboard/file export.

1-Click Overleaf Chrome Extension (Manifest V3)

  • Background service worker and custom popup.
  • Content script targeting *://*.overleaf.com/project/* that interfaces directly with Overleaf's CodeMirror 6 DOM (.cm-content) to inject compiled LaTeX at the cursor position, backed by automated toast notifications.

Challenges I ran into

  • Handling Real-World Classroom Artifacts: Generic OCR models fail completely on classroom photos because perspective angles distort text height, and ceiling lights wash out ink. Building the OpenCV preprocessing pipeline (homography unwarping and CLAHE equalization) was essential to make the multimodal vision model reliable.
  • TikZ Vector Synthesis: Getting the AI model to consistently generate valid, compilable TikZ code without syntax bugs required strict prompt schemas, isolated coordinate bounds, and explicit package definitions (\usepackage{tikz}).
  • Overleaf DOM Injection: Overleaf runs modern CodeMirror 6, which uses a virtualized DOM and internal document state models. Standard textarea.value assignments do not work and cause state desynchronization. I implemented DOM range insertions and synthetic input event dispatchers to preserve editor history and undo stacks.

Accomplishments that I am proud of

  • Beyond a Thin Wrapper: Built a legitimate computer vision pipeline with OpenCV homography, CLAHE glare removal, and layout segmentation that cleans physical image inputs before passing them to the AI.
  • True Vector Graphics: Generating actual, editable LaTeX TikZ code from rough, hand-drawn whiteboard sketches instead of low-resolution raster images.
  • Frictionless Workflow: Eliminating the download-copy-paste cycle entirely through the Overleaf Chrome extension.
  • Solo Execution: Architected, built, and tested the backend, computer vision pipeline, frontend studio, and Chrome extension independently.

What I learned

  • How to construct perspective transformation matrices and manage color spaces in OpenCV.
  • The mechanics of CodeMirror 6 DOM internals and Manifest V3 Chrome extension message passing.
  • How to structure multimodal prompts to generate production-quality vector graphics (TikZ) rather than simple text approximations.
  • In-memory PDF rasterization and stream management with pypdfium2.

What's next for Notes2Tex

  • Bi-directional Sync: Allowing users to highlight LaTeX code in Overleaf and see the corresponding section of the original whiteboard photo.
  • Live Classroom Audio Pairing: Synchronizing recorded lecture audio with whiteboard snapshots to generate timed section headings and proof explanations.
  • Equation Step-by-Step Solver: Adding an optional proof-verification assistant to highlight potential arithmetic mistakes in handwritten derivations.

Built With

Share this project:

Updates

Submission history