Archie

Archie!
Pen holder
Motion device for pen
Graph + initial sketch

Inspiration

We wanted to fix a small, universal frustration: nobody can draw a straight line or a clean circle freehand. The first idea was a wearable exoskeleton that nudges your hand while you draw. Working through the control problem (backdrivable motors, force-vs-position control, clutches) we realized that safely force-correcting a human hand in 36 hours was the wrong fight. So we flipped the architecture: instead of correcting you while you draw, the system watches you demonstrate, figures out what you meant, and then a robot redraws it perfectly. Show it your intent; it makes the intent real. This is mostly meant for professionals such as architects during their sketches on their day job, or for lessons or practice for surgeons to understand precise movement.

What it does

Archie is a learn-and-reproduce drawing platform with two devices:

sensor pen: two optical-flow sensors (salvaged from computer mice) plus an MPU-6050 IMU on an ESP32 streams motion deltas and pen up/down as you draw.
Host software reconstructs the trajectory in millimetres, segments strokes, and classifies intent: was that wobbly loop a circle, a rectangle, a polyline, deliberate freeform, or something else?
The SCARA robot arm (two MG90D servos, ESP32, acrylic linkage) then redraws the idealized result.

A web studio dashboard shows live sensor health, calibration status, and reconstruction confidence at every stage.

How we built it

The pipeline is: raw sensor deltas → calibration → sensor fusion → stroke reconstruction → intent classification → cleanup → motion planning → robot.

Calibration. Optical mouse sensors report ticks, not millimetres. We fit a full 2×2 affine matrix per sensor from controlled translation runs, so axis scale and cross-axis skew are both recovered:

$$\mathbf{p} = A\,\mathbf{c}$$

A calibration-lab module captures rig sessions, fits candidate profiles, and because we didn't trust ourselves, validates every candidate against held-out sessions with a manifest-hash leakage guard before it can ever be activated (thresholds: ≤2 mm error per 100 mm translation, ≤2° per 90° rotation).

Reconstruction. Two spaced sensors let us solve rigid-body motion: differential displacement between them gives rotation, fused with the gyro via confidence-weighted filtering. The solver degrades gracefully where single-sensor fallback and gyro carry are explicit, opt-in, and reported in a sensor-status telemetry stream instead of silently pretending everything is fine.

Intent. Each stroke gets fitted against candidate models. Lines via SVD, circles via linear least squares:

$$\min_{c,\,r}\sum_i \left(\lVert p_i - c\rVert - r\right)^2$$

Candidates are scored with residual-based confidence s = exp(−k·ε/scale), ambiguity is preserved and surfaced rather than hidden, and cleanup runs under named profiles (FAITHFUL / SMOOTH / PRESERVE_DETAIL) that report exactly how far the cleaned path deviated from what you actually drew.

Stack: ESP32 firmware, Python host (FastAPI, pydantic v2 frozen domain models, numpy), CRC-checked serial protocol with sequence numbers, web dashboard, 131 unit tests, and a golden end-to-end demo fixture that replays deterministically.

Challenges we ran into

Our synthetic calibration data was lying to us. Per-frame sensor counts were being rounded independently, biasing the ground truth itself; fits looked worse than the algorithm deserved. Fixed by redistributing counts Bresenham-style so truncation error can't accumulate.
Freeform is the hardest class. Our classifier hits 86.7% on the labeled corpus but structured shapes (line/polyline) consistently outscore genuine freeform scribbles, shape fits are "greedy." Known open problem.
Motor selection. Hobby servos can't be backdriven; backdrivable motors can't hold position. The pivot from force-nudging to capture-then-reproduce came directly from losing this argument with physics.
Identifying salvaged silicon. Pulling optical-flow sensors out of donor mice means datasheet archaeology for whatever chip you find.

Accomplishments that we're proud of

A deterministic replay system. Every recorded session reproduces byte-identical results, so algorithms improved without re-doing physical demos.
131 tests green, strict mypy, mid-hackathon.
The full pen→reconstruction→intent→plan→robot pipeline runs as one golden demo.

What we learned

Cheap mouse sensors can do millimetre-scale tracking, but only with real calibration, cross-axis terms and per-direction hysteresis are not optional.
Report honest error, not flattering error. Every time we made a metric more honest, it found a bug.
Separating "what did the user mean" from "how does the robot move" made both problems tractable.

What's next for Archie

Training an intent model on our own handwriting with Freesolo's post-training platform: SFT warm-start, then GRPO where the reward is literal geometric accuracy: the model gets scored on how close its claimed shape is to the ground-truth points.
Extended shape vocabulary: n-gons, lower-case letters, everything which a human can draw or paint can be done with Archie.
Semantic text mode: write a word sloppily, get it back in a perfect single-stroke font.