TetrisDog: A Physical AI Education Kit (GPT-5.6)

Watch human intent become safe robot motion in real time — GPT-5.6 reasons, A* plans, and a human operator confirms every move.

Comment

Inspiration

AI education is often trapped between abstract theory and inaccessible hardware. Students can write code in a simulation without ever seeing how an intent becomes a safe physical action, while real robotics platforms are often too expensive or complex for a first experiment.

The spark came from something unexpectedly simple: play the grid-jumping game at Pixel Room ( C.H.E Arcade, Bellevue ,WA, USA). Choosing where and when to jump turned a familiar grid into a system of spatial decisions — each square a possible state, each jump an action over time, each blocked square changing what could happen next. That image stuck as I thought about robot learning: how a robot occupies space, how its state changes moment to moment, and how obstacles reshape the relationship between perception, planning, and action.

TetrisDog turns that intuition into an interactive learning environment — a Tetris-inspired spatial sandbox where learners can see a robot's world, not just read about it.

What it does

TetrisDog is a Physical AI education kit that makes the normally hidden pipeline — from human intent to robot action — fully inspectable.

Learners place obstacles on a live 3D grid and set a goal for an AI "dog." They express intent through touch, text, or voice. A constrained GPT-5.6 layer classifies a perception event into one of four allowlisted grid intents (CONTINUE / STOP / REQUEST_REPLAN / MARK_OBSTACLE) — it never issues motor commands directly. A deterministic A* planner computes (and replans) the route whenever the grid changes. The learner inspects the plan, confirms it, and watches a simulated trajectory execute. Every run is recorded as a structured observation-action-reward JSONL episode, seeding future robot-learning research.

The result: instead of one opaque black box, learners see perception, semantic reasoning, planning, human authority, and outcomes as separate, inspectable stages — each with its own role and failure mode.

How we built it

I split the system into independent, testable layers:

Python logic engine (asyncio + websockets) coordinating state, planning, clients, and hardware endpoints Constrained semantic layer: GPT-5.6 via the OpenAI Responses API with Structured Outputs, mapping perception descriptions to a strict intent schema — with a transparent local rule as offline fallback A* global planner producing a deterministic reference route through the occupancy grid Three.js + Vite frontend rendering the grid, obstacles, agent, and path in real time A versioned WebSocket JSON protocol ("neural highway") moving world state, planning requests, trajectories, and execution acknowledgements between every component A dry-run-only hardware adapter, with real motor control kept behind an explicit, separate safety boundary Modular sensor scaffolds (BLE proximity, OAK-1/AprilTag) that publish validated observations without ever commanding motors

Where Codex accelerated the build: Codex helped us prototype the Three.js + WebSocket interaction loop fast enough to go from a static grid to a touch/text/voice-controlled demo within our MVP window; it helped cleanly separate the planner, world model, protocol, and frontend into components learners can actually inspect and debug; it helped us define and test the async JSON message flow end-to-end; and it kept our architecture docs, setup instructions, and tests aligned as the prototype changed under time pressure.

Where GPT-5.6 is used: strictly as a constrained semantic reasoning layer, not a controller. It only ever outputs one of four allowlisted grid intents from a perception description — it cannot emit motor, PWM, or velocity commands. Every recommendation, its source, and the human's response to it are logged, making semantic assistance measurable instead of invisible.

Challenges we ran into

Keeping GPT-5.6 strictly bounded to a safe intent schema — resisting the temptation to let it "help more" by touching planning or execution Designing a protocol expressive enough for multimodal input (touch/text/voice) and multi-device output (browser, Rokid AR) without coupling everything directly to hardware Being honest about scope in a 10-hour build: deciding what stays "future work" (live OAK-1 capture, Q-learning, production robot adapters) versus what needed to be real today (the interaction loop, the protocol, the safety boundary)

Accomplishments that we're proud of

A fully runnable, safe, human-in-the-loop loop — from multimodal intent to inspectable AI reasoning to deterministic planning to recorded outcome — built and working within the Build Week window

A clean architectural separation between perception, semantic reasoning, planning, human authority, and hardware, so each piece can be studied, tested, or replaced independently Every interaction is captured as structured episode data, turning a classroom demo into a real seed dataset for future embodied-learning research

What we learned

Separating "what the AI recommends" from "what actually happens" — and logging both — is one of the clearest ways to teach AI literacy. It also turned out to be the right engineering pattern for safety: constraining GPT-5.6 to a narrow, auditable intent schema made the whole system easier to reason about, test, and trust.

What's next for TetrisDog:

Data loop ✅ Live OAK-1 + AprilTag capture validated on real hardware (calibration, timestamps, confidence, sensor health) Synchronized RGB-D, 6DoF, tactile, force, and telemetry schemas Unified, replayable episode records; publish 3–5 real-world episodes Export PIER-style / LeRobot-compatible datasets

Planning → learning Tabular Q-learning alongside A*; behavior-cloning baseline from demonstrations Richer world model (terrain, uncertainty, dynamic obstacles) Compare planning vs. semantic assistance vs. learned policies on success rate, intervention rate, latency

Platform Safety-gated adapters for ESP32, quadruped SDKs, RGB-D, VIVE Trackers WebXR/AR adapters (Rokid, Quest, Aria Gen 2); sim-to-real validation

Education Predict–reveal–compare checkpoint; accept/reject/correct before execution Classroom/workshop data-collection sessions; consent & dataset templates

Built With

Share this project:

Updates