Cardboard Cockpit

Inspiration

Everyone who has ever wanted to fly a fighter jet runs into the same wall: the hardware tax. A decent HOTAS setup costs $300–$500. A VR headset costs more. Flight sim is one of the oldest genres in gaming and it is still gated behind a pile of plastic that most people will never buy.

Meanwhile, the highest-bandwidth input device most of us already own is a webcam — and it sits there doing nothing but video calls.

So we asked a stupid question: what is the cheapest possible cockpit? The answer we landed on was a sheet of paper. Print some markers, fold a yoke, hold it up, and let the camera do the work. Cardboard is the only material on Earth that costs nothing, is infinitely reconfigurable, and that everyone already has in a recycling bin.

And once we had a jet, it needed something to shoot at. We went with a giant goose.

What it does

Goose Protocol — SPECTRE X-26 is a native desktop flight simulator you control with printed cardboard and a webcam. No drivers, no VR, no controller.

The cardboard yoke owns pitch, roll and both latched weapon switches
The cardboard throttle owns power, boost and braking
A wireless BLE badge handles secondary controls — start, pause, gear, assisted landing, camera views
Keyboard and mouse always work, and always take priority the instant you touch them

You fly a real, photogrammetrically sourced San Francisco — SFO, downtown, the residential grid, the hills, the Golden Gate and Bay Bridges, Alcatraz — on a mission that escalates from routine intercepts to an anomalous signature, and then to a 16,000-HP boss goose with staged weak regions.

The entire demo — takeoff, city run, escalation, boss, aftermath, assisted landing at SFO — is capped at 150 seconds. Unlimited ammunition, forgiving flight assistance, spoken in-mission coaching that hands control back to you after the first three contacts, and crash recovery that puts you back into stable flight instead of a loading screen. It's an arcade demo, not a certified simulator, and we're upfront about that.

For the full setup, a laptop camera reads the yoke while a phone (over Continuity Camera) reads the throttle — two cameras, two independent trackers, two live preview windows.

How we built it

Three layers, talking over one local socket.

The vision layer — Python + OpenCV

A local ArUco tracker using the DICT_4X4_50 dictionary. Marker 7 is the yoke; 0 / 1 / 2 form a relative three-tag throttle (idle, slider, full) so the throttle self-calibrates from live geometry instead of absolute pixel positions; 31/32 and 41/42 are flip-tab weapon switches.

Estimating the yoke's orientation from a single planar square is the hard part, because SOLVEPNP_IPPE_SQUARE returns two valid poses for a planar marker — a genuine mathematical ambiguity that gets worse the closer you are to head-on. We resolve it by scoring both candidates on reprojection error with a small temporal-continuity bias toward the previous frame's pose:

$$ \text{score} = \underbrace{\sqrt{\tfrac{1}{4}\sum_{i=1}^{4}\lVert \pi(\mathbf{R},\mathbf{t},\mathbf{X}i) - \mathbf{u}i \rVert^2}}{\text{reprojection error}} ;+; 0.01 \cdot \min!\big(|\Delta\theta{\text{prev}}|,, 45^\circ\big) $$

and reject the frame entirely if the winning error exceeds (4.0) px. Confidence is then a product of apparent marker size and fit quality:

$$ c = \operatorname{clamp}!\left(\frac{s}{85},, 0.25,, 1\right) \cdot \operatorname{clamp}!\left(1 - \frac{\varepsilon}{6},, 0,, 1\right) $$

Duplicated marker IDs are treated as ambiguous and rejected rather than resolved by picking whichever one OpenCV enumerated last. Optional checkerboard lens calibration replaces the approximate (f \approx 0.72,w) pinhole model with measured intrinsics and distortion.

Every frame stays in memory. Nothing is recorded, nothing is uploaded.

The transport — a hardened local WebSocket

This is where we spent a disproportionate and, in hindsight, completely justified amount of effort. The game speaks to the tracker over ws://127.0.0.1:8765 with a versioned JSON protocol, and the client treats every packet as hostile:

2 KB packet cap, strict per-field type checks, finite-number and JSON-safe-integer validation
Nothing mutates state until every field validates — malformed data must not refresh the staleness timer, advance the sequence counter, or momentarily override the pilot
Monotonic sequence numbers reject replays and out-of-order frames
A reconnect resets confidence to zero: a fresh handshake cannot make old control values trustworthy again
A 350 ms staleness window, after which the yoke eases toward neutral rather than snapping
Latched weapon switches force-release on signal loss, so you don't lose the camera and keep firing forever
Yoke and throttle confidence are tracked independently, so a dropped phone connection holds your last throttle setting while laptop steering keeps working

The simulator — Godot 4.7.2, Forward+ / Metal

The flight model is deliberately fly-by-wire fiction, not aerodynamics — tuned for readability, not fidelity. Physics runs at a fixed 60 Hz tick, subdivided further so that behavior is identical regardless of frame rate:

$$ n = \max!\left(1,; \left\lceil \min(\Delta t,, 0.5) \cdot 120 \right\rceil \right) $$

All control response uses exponential decay rather than naive lerp, which makes smoothing genuinely frame-rate independent:

$$ x_{t+1} = x_t + (x_{\text{target}} - x_t)\left(1 - e^{-k,\Delta t}\right) $$

Banking produces yaw through a coordinated-turn approximation, so the jet turns the way players expect rather than the way a real airframe would:

$$ \dot{\psi}{\text{coord}} = \frac{\sin(\phi)\cdot F{\text{bank}}}{\max(v,, 55)}, \qquad F_{\text{bank}} = 64 $$

Lift and sink are a speed ratio against rotation speed, with a bank penalty:

$$ L = \operatorname{clamp}!\left(\frac{v}{0.82, v_{\text{rot}}},,0,,1\right), \qquad \text{sink} = 28(1 - L) + 6\big(1 - \max(\cos\phi,,0)\big) $$

Drag combines an induced term with configuration penalties:

$$ D = a\left(\frac{v}{v_{\max}}\right)^{2} + \begin{cases} 0.70 & \text{gear down} \ 0.15 & \text{gear up}\end{cases} + n_{\text{flap}},(0.35 + 0.007,v) $$

Landings are graded, not pass/fail:

$$ \text{score} = \operatorname{clamp}\Big(100 - 7|\dot h| - 0.8|\phi^\circ| - 0.45,d_{\text{center}} - 0.7\max(0,, v - 1.15,v_{\text{rot}})\Big)_{0}^{100} $$

Crucially, flight_dynamics.gd is a plain RefCounted with zero engine dependencies — which is what made everything below possible.

Every tunable constant in the game — 140 of them — lives in a single balance.gd with documented provenance for each value.

Hardware and assets

An ESP32 badge over BLE supplies secondary controls with phase feedback, auto-retry and stale-input release. Assets are real: licensed FlightGear airframe sources kept byte-identical, sourced SF aerial photography and terrain baked into LOD'd native meshes, and radio voices built from licensed Kenney recordings run through band-pass filtering, compression, saturation, RF hiss and PTT squelch — processed human audio, not generated voice clones.

Challenges we ran into

The planar pose ambiguity nearly killed the project. Holding the yoke flat toward the camera is the most natural thing a player does, and it's exactly the configuration where IPPE's two solutions are least distinguishable. Early builds would flip pitch sign at random and roll the aircraft inverted for a frame. The fix was the reprojection-plus-continuity scoring above, plus a PoseGuard that rejects physically implausible jumps.

Camera input is unreliable by nature, and games assume input is not. Your hand leaves frame. Glare washes out a marker. The phone drops off Wi-Fi. Every one of those was, at first, a crash into a hillside. We ended up rewriting the input client three times before landing on the model that actually works: confidence is a first-class value, staleness is a first-class value, and the absence of input is a defined state rather than a stale value nobody cleared.

Two cameras means two independent failure domains. The moment we split yoke and throttle across a laptop and a phone, one global tracking boolean became a bug. Confidence had to be split per-control, and the status line had to express four states, not two — including "throttle tracked, keyboard steering," which is a real thing that happens constantly.

You cannot playtest a webcam in CI. A physical control scheme is exactly the thing an automated test can't touch. Solving this shaped the whole architecture.

Repository weight. Real photogrammetry is enormous. We ended up with ~1.5 GB across 5,300+ files, of which ~17,000 lines are actual code and the rest is terrain, meshes and audio. Git is not happy about this and we know it.

The 150-second constraint. Fitting takeoff, a city run, escalating encounters, a boss fight and a landing into two and a half minutes meant cutting the long return leg entirely and replacing it with a transition — which then had to not feel like a cheat.

Accomplishments that we're proud of

The input path is engineered like a safety boundary, not a game feature. Validate-before-mutate, replay rejection, independent confidence, graceful decay to neutral, forced release of latched state. We're fairly confident this is more input hardening than most shipped commercial games have, and it's the reason the thing is actually playable rather than a demo that works if you hold very still.

We made a camera-controlled game testable without a camera. Because the flight model is engine-free and the vision layer speaks a documented socket protocol, our suite runs 27 headless Godot test scripts plus Python unit tests plus complete simulated sorties — full takeoff-to-landing runs, malformed-packet handling, tracker dropout and reconnect — on GitHub Actions, with no webcam involved. A 460-second simulated SF round trip hitting 14/14 waypoints and stopping safely on runway 28R is a CI check.

Graceful degradation that a player never notices. Drop the marker mid-turn and the aircraft eases to level while the HUD quietly says YOKE LOST · KEYBOARD READY. You just keep flying. That invisible behavior took more engineering than the boss fight.

We documented what we did not verify. Our verification notes label FPS figures as observed samples rather than guarantees, and explicitly list what remains untested: physical cardboard operation at scale, Intel runtime behavior, other platforms, Apple notarization. Honest evidence beats an impressive-sounding claim.

It's genuinely fun, which was never guaranteed. Cardboard could easily have been a gimmick that you try once. Banking a jet through the Golden Gate by tilting a folded piece of paper is not a gimmick.

What we learned

Confidence is not a boolean. The single biggest design insight of the project. "Is the marker visible" is the wrong question; "how much should I trust this, right now, for this specific control" is the right one. Everything good in our input layer follows from that.
Absence of data is a state that must be handled explicitly. Nearly every early bug was some variant of stale value nobody cleared. A reconnect resetting confidence to zero — refusing to let a handshake relaunder old trust — is a one-line idea that eliminated an entire bug class.
Exponential smoothing, not lerp(a, b, 0.1). The (1 - e^{-k\Delta t}) form is the difference between a game that feels identical at 30 and 120 FPS and one that doesn't. We should have started there.
Testability is an architectural decision made on day one, not a chore added at the end. Keeping the flight model free of engine types felt pedantic in hour three and saved the project by hour thirty.
Physical interfaces fail in ways software interfaces don't. Lighting, glare, focal length, hand tremor, how far someone naturally sits from a laptop. None of it is in the code and all of it determines whether the thing works.
Centralize your tuning. One balance.gd with every constant and a comment explaining each value's origin made rebalancing a five-minute job instead of an archaeology expedition.

What's next for Goose Protocol

Ship it lighter. Move the source archives and heavy geometry to Git LFS or release artifacts. A 1.5 GB clone is a barrier to every contributor.

Get off macOS. The tracker is portable Python and Godot is cross-platform; the blockers are the Metal renderer path and the Apple Silicon badge helper. Windows and Linux builds are next, along with proper notarization.

Markerless hand tracking. ArUco is robust and cheap, but printing markers is still friction. A hand-pose model that infers yoke orientation from the hands themselves would drop the setup cost to zero.

More sky. The engine already retains coastal and alpine regions as regression fixtures; exposing them as playable maps is mostly a menu problem.

Calibration that teaches itself. The guided checkerboard flow works, but the ideal version watches you hold the yoke for five seconds and infers everything — neutral pose, lens intrinsics, comfortable range of motion.

Open the protocol. The WebSocket schema is versioned and documented. We'd like other games to be able to read a cardboard yoke — the control scheme shouldn't belong to one title.

Multiplayer. Two people, two laptops, two pieces of cardboard, one sky.