Redline — the robot that marks your paper

Inspiration

The red pen is the most reused tool in education. A teacher with thirty essays makes the same marks thirty times — recieve, alot, should of — and the student still has to hunt for where each mark is aimed.

The other half of the idea was a gap: a language model can proofread a page in seconds, but it cannot touch the paper. The model thinks in text coordinates ("line 3, character 10"); the world thinks in inches. Redline is the translator between them.

What it does

Hold a page under the camera and say "hey Redline, check this text." Redline photographs it, has Gemini proofread it, reads every mistake and its correction out loud, and prints where each one sits, in inches from the corner of the sheet:

1. line 1 char 1   "i"       -> "I"        capitalization   1.05 in from the left, 1.24 in down
2. line 1 char 9   "its"     -> "it's"     punctuation      1.92 in from the left, 1.24 in down
3. line 1 char 21  "recieve" -> "receive"  spelling         3.21 in from the left, 1.24 in down

It also saves a marked copy (last_marked.jpg) with a red box on every mistake. Saying its name alone is a command, and "goodbye" signs it off.

How we built it

Four moving parts. OpenCV takes the picture: it finds the sheet, flattens it and measures the ink. Gemini does the reading, returning every mistake as structured data — the wrong word, the correction, the type of error, and the line and character where it starts. ElevenLabs does the voice, listening with Scribe and narrating with a streaming voice. And layout.py joins the two halves, turning a text position into (x, y) in inches on the page:

$$ x = m + \left(c - \tfrac{1}{2}\right) p_\ell, \qquad y = \frac{t_\ell + b_\ell}{2} $$

where t^ell, b^ell are the measured top and bottom of line ell's band of ink and p_\ell is the pitch recovered from that line — so a ragged-right line still lands on the right word. We never ask the model for coordinates: counting characters in its own transcript is something it is good at; guessing inches is not.

Challenges we ran into

Indices are not coordinates. Assuming a monospace grid, character c at works on flat text and falls apart on a photo of proportional type. Measuring the real line bands and deriving the pitch per line fixed it.

Perspective. A fixed millimeter-to-pixel factor is wrong everywhere but the center. Detecting the sheet as a quad and warping it to a top-down A4 or Letter page removed the problem — but dark desks, shadows and page edges kept fooling the contour search, so we now trim the border and require a minimum ink density before accepting a sheet. With no sheet at all, as in a screenshot, it treats the picture as the page.

Latency you can hear. A photo plus a model round trip is seconds of dead air, so it narrates in character while it works and runs speech on a background thread.

Accomplishments that we're proud of

• It knows where, not just what. On a photo of a page tilted onto a desk, the worst of seven planted mistakes lands 0.052 in from its true position — under half a character cell (0.11 in) — and the test asserts that every point sits on dark ink, not in the gap beside the word.
• Measured, not assumed. From the photo alone it recovers the page's line spacing exactly (0.34 in) and a character pitch of 0.099 in.
• It runs with no hardware and no API keys. A stubbed microphone and a generated page of known geometry mean the whole pipeline is testable before the camera exists.
• It is hard to break on stage. Mangled names still trigger it, stray sentences do not, and a busy model is absorbed by the fallback chain.
• The arm cannot be sweet-talked. The model may only call four named actions, never raw angles, and every target is clamped to joint limits with a per-turn action budget.

What we learned

The hard part of an AI that acts in the world is not the AI — it is the coordinate frames. Tokens become text positions, positions become inches, inches become joint angles, and every hop is a chance to be quietly wrong. Asking the model only for what it is good at, and doing the geometry ourselves in plain code, was the best decision in the project.

Building the fixture before the feature turned "it looks about right" into a number, and a red box on the wrong word into a failing test.

What's next for Redline

• Let the arm point. The inch map exists for this: convert page inches into the arm's frame and hand the target to the arm's existing point_at action, so the demo ends in a physical gesture.
• A camera over a desk, so the page never has to be held up.
• Handwriting mode for grades 1–5: letter formation, not just spelling.
• Per-student history, so it stops re-teaching the same word.

Boring work, automated.

Built With

• computer-vision
• elevenlabs
• git
• github
• google-gemini
• hardware
• numpy
• opencv
• python
• speech-to-text
• text-to-speech
• webcam