Inspiration
Generative systems can propose compelling visual ideas, while plotters can execute precise motion, but connecting them directly creates an unacceptable control boundary. Embodied Plotter Agent asks whether AI can interpret artistic intent while deterministic software retains complete authority over physical execution.
What it does
A user enters a natural-language drawing request. GPT-5.6 returns only a typed specification made from a restricted primitive library. Deterministic code generates and optimizes vector paths, renders an SVG preview, simulates motion, enforces geometric and operational limits, emits Klipper-compatible G-code, and validates that G-code again with a whitelist parser. After explicit human approval, Moonraker uploads the file and Klipper executes it.
Two Orange Pi cameras show the canvas and pen tip. Local OpenCV rectifies the canvas, estimates progress, and compares observed ink with the planned path. GPT-5.6 receives only structured observations or selected snapshots, never raw video. A final report records the plan, validation, execution, and vision estimates.
How we built it
Python, FastAPI, Pydantic, NumPy, OpenCV, HTTP/WebSockets, plain HTML/CSS/JavaScript, Klipper, Moonraker, and systemd run across an Orange Pi 5B and an ATOL touchscreen terminal. GPT-5.6 is isolated behind strict JSON Schema output. Geometry, safety, G-code, physical authorization, and execution remain deterministic.
The repository includes real hardware mode and a one-command judge mode using simulated Moonraker and synthetic fiducial camera frames. Run ./run_demo.sh or .\run_demo.cmd, then open http://127.0.0.1:8000. No plotter, camera, API key, or account is required; cached GPT output is clearly labeled while all deterministic and OpenCV stages run locally.
How Codex and GPT-5.6 were used
Codex was the principal engineering collaborator throughout Build Week. It inspected both Linux/Klipper installations, designed the Orange Pi/ATOL architecture, created backup-first migration and rollback procedures, implemented the strict schema and deterministic drawing/G-code pipeline, built Moonraker and OpenCV integrations, added the recording UI and tests, diagnosed USB camera throughput and MCU reconnects, recovered KlipperScreen, calibrated the real canvas transform, and prepared the release.
GPT-5.6 performs only creative reasoning: interpreting intent, choosing deterministic primitives, producing the typed drawing specification, proposing schema-level repairs, and interpreting aggregated observations. It never controls the MCU, emits raw G-code, runs shell commands, changes Klipper configuration, or overrides safety. Every run records the exact model and whether the response was live or cached.
Challenges we ran into
The hardest work was the boundary between creative reasoning and physical safety: closing every model output behind a typed schema, transforming logical canvas coordinates into the verified machine frame, preserving the legacy touchscreen, keeping two cameras usable, and ensuring stale generated files cannot survive calibration changes. Independent validation, job-bound approvals, a second G-code parser, fail-closed gates, backup-first deployment, and deterministic pen-up calibration address those risks.
Accomplishments that we're proud of
- A real prompt-to-Klipper pipeline where GPT-5.6 never emits G-code.
- Deterministic geometry, travel optimization, simulation, and two-stage validation.
- Real Orange Pi/ATOL/Klipper/Moonraker integration with two live cameras.
- Local OpenCV planned-versus-observed analysis.
- Reversible deployment with no MCU reflash or automatic printer configuration changes.
- A one-command hardware-free demo and 56 automated tests.
What we learned
Embodied agents are most convincing when intelligence and authority are deliberately separated. The language model is valuable for intent and interpretation; deterministic software is better for geometry, policy enforcement, and physical control. USB topology, stale-job integrity, camera latency, and rollback design matter as much as model quality when software touches a real machine.
What's next
Improve calibrated ArUco accuracy, add controlled recovery from temporary camera loss, characterize line-width and pen-position uncertainty, expand the safe parametric library, and package the system for other Klipper-based drawing machines.
Log in or sign up for Devpost to join the conversation.