Can an Agent Fly a Plane?

Inspiration

I've been watching a lot of pilot content lately, and I've always been interested in planes. An agent-driven flight evaluation sounded cool, and using STL, where I'm from, felt obvious.

What it does

This project is a real-time WebMCP evaluation for model developers and evaluation researchers. Your agent has to take off from STL on a flight to MDW in Chicago, handle a sealed emergency, coordinate a diversion with simulated ATC (air traffic control), and land safely back in St. Louis.

ATC defines the route and checkpoints, but the agent decides how to fly them. It chooses the flight targets, aircraft configuration, and activation conditions. The simulator applies those targets continuously using 60 Hz physics at a fixed 2x time scale. It never pauses or changes that rate while the agent thinks, and the emergency decision window and mission deadline still use real elapsed time.

Completing the flight requires hitting the route checkpoints and configuring the aircraft correctly. The score deducts points for slow emergency decisions, exceeding the mission window, high G-force, abrupt motion, hard or off-center landings, and crashes.

The page registers six WebMCP tools for reading the manual, starting the flight, programming commands, coordinating with ATC, and waiting for events. WebMCP gives the agent structured state and validated actions instead of making it guess at visual controls. A person can fly the same aircraft manually, or start an agent evaluation and watch it go. The exported trajectory records every WebMCP call, action, result, latency, before-and-after state, score change, and simulator event.

How I built it

I used Codex to build and test each version. My first Cesium.js scene used a real route from STL to Chicago, but it hurt performance and was very buggy. I switched to Three.js and built a simplified but representative wide-body twinjet flight envelope.

The simulation runs outside React render state at 60 Hz. The WebMCP layer uses document.modelContext.registerTool to expose the six tools, but the simulator still works manually without WebMCP if you want to fly it yourself.

I went through probably 15 to 20 iterations of the world itself, adding audio and the minimap along the way. The hard part was deciding what the agent should actually control. We revised the tools until the agent was calling the shots instead of taking credit for decisions buried in the simulator.

Challenges I ran into

The V1 version was kind of cool, but it was too easy. The agent was basically on autopilot. The deterministic simulator controlled pitch, bank, and power while the agent only decided whether to return or continue.

I set a /goal challenging Codex to make a fresh subagent complete the flight using the actual flight model. The results were pretty interesting.

After 13 hours and 73 attempts, it finally got three runs to land with scores around 90, but of course it cheated. Somewhere along the way it decided the easiest way to fly a plane was to stop time whenever it needed to think. That obviously defeated the entire point.

That exposed the most interesting part. As soon as the agent has to think, time becomes part of the problem. Even Terra at low reasoning could take long enough that the aircraft crashed or flew too far away.

The solution was to meet in the middle. Simulated ATC supplies the route, but the agent writes the flight program before takeoff and replaces it after the emergency. The simulator tracks its targets at 60 Hz without pausing or deciding what comes next.

I tested that version with ten agents in cold threads, and all ten failed. I decided they needed a “manual” explaining the aircraft and command interface. I really didn't want to do this, but they needed enough context to land without moving the decisions back into the simulator.

The manual publishes rotation speed, climb and landing configuration, approach speed, flare limits, and the go-around sequence. It does not reveal the emergency, choose the diversion, or provide a program. The agent still has to fly.

The resulting architecture completed three fresh blind runs in a row without pausing, changing the fixed simulation rate, or revealing the emergency early. Every WebMCP interaction was captured in the browser-side trajectory export.

Accomplishments that I'm proud of

I'm proud that I didn't stop at “this works because the agent said it does” (which happens every single time lmao). I grilled Codex on how the simulation worked and kept pushing until the agent, rather than a hidden policy, owned the decisions.

Thinking still hurts performance here, but I believe that is useful signal rather than something to hide. Models should be able to judge when deeper reasoning is worth the time.

What I learned

I learned a lot about building with WebMCP and understanding the agent experience. It isn't as simple as “give the agent all the information and let it do the thing.”

I had to run fresh agents repeatedly and fix ambiguous tools without taking decisions away from them. The manual is part of the interface, not an answer key.

The same applies to human UX. If people and agents get stuck at the same point, the interface probably is not communicating clearly enough.

What's next?

This could become an evaluation environment for model developers and researchers testing decisions under time pressure.

Most evaluations I've seen analyze time after the run. Here, time changes the environment. A longer response means the aircraft has traveled farther, burned more fuel, and moved closer to failure. The sealed scenario and exported trajectory make that measurable.

The idea also does not have to stop at flying a plane. The same setup could test an agent acting as ATC, coordinating a ground crew, or handling another system where the world keeps moving while the model thinks. The bigger question is whether models can learn when to reason deeply and when they need to act now.

Built With

• codex
• three.js
• typescript
• webmcp