Unstep

A second set of ears for every air traffic controller: AI agents that catch missed calls and wrong readbacks, with the controller in command

Comment

Inspiration

Every flight you take is guided by a voice on the radio: an air traffic controller. The FAA handles about 45,000 flights a day, and the people doing it are stretched thin.

  • The FAA is about 3,500 controllers short of its own goal, with many on mandatory overtime and six-day weeks.
  • Washington Reagan National runs on 20 certified controllers where it should have 30.
  • At DCA in January 2025, one controller was working two positions at once. As Transportation Secretary Sean Duffy put it on ABC: "There is a level of risk that gets injected into the system when we have a controller that's doing two jobs instead of one."

Then we learned how fragile the radio itself is. Pilots and controllers share one half-duplex channel: only one voice gets through at a time. When two people transmit at once, everyone else hears a squeal, and neither speaker knows. An FAA study found blocked or clipped calls in 1.16% of transmissions at the busiest US approach facilities. At the NTSB hearing on the DCA midair, the tower operations manager was asked whether a controller can tell that a call was stepped on. His answer: "There's nothing."

At a busy facility handling around 2,000 transmissions an hour, that rate means

$$ 2000 \times 0.0116 \approx 23 \text{ blocked or clipped calls every hour,} $$

each one a chance that a pilot never heard "hold short". An overloaded controller can't catch all of them alone. We didn't want to replace that controller; we wanted to give them a second set of ears.

What it does

Unstep is a team of AI agents that works the radio with the controller. The controller approves everything; the agents never act alone.

Job What the agents do
Listen Turn every transmission, tower and cockpit, into live text
Catch Detect a stepped-on call and recover the words that were lost
Check Compare every readback with the instruction; a wrong runway is flagged and the aircraft doesn't move until it's right
Draft Propose the next clearance from the flight plan, the aircraft's last call and live runway occupancy
Speak Say the recovered instruction privately to the one pilot who missed it, in the left ear, in their language
Record Log every decision for audit and write the shift briefing

The demo: the laptop is the control tower, and phones are the planes. Scan the QR code and you're a pilot. Here's one moment from it:

  1. The tower says "Cessna 1-2-3, hold short runway 33" while the helicopter keys up over it.
  2. Unstep catches the overlap and recovers the instruction from the Cessna's own memory.
  3. It cites the FAA rule.
  4. The controller approves with one tap, and only the Cessna hears the fix.
  5. The pilot reads it back, and Unstep checks it: "Read back correct."

How we built it

Cartesia: the ears and the voice. On a radio, everything is voice.

  • Ink-2 streaming speech-to-text transcribes the tower and every cockpit while they talk. Each cockpit hears its own live mix of the frequency. The transcript is final 155 ms after the mic is released (it was 2.7 s before we streamed).
  • Key-term prompting with 39 aviation terms per cockpit: callsigns, runways, taxiways, "hold short", "niner".
    • In heavy simulated radio noise, callsign accuracy went from 62.5% to 95.5% and runway accuracy from 81.6% to 98.5%.
    • Word error rate went from 0.191 to 0.009.
  • Sonic-3.6 speaks the private advisory, streamed from the first chunk, with first audio in 85–201 ms.
    • It uses context cancel when the pilot keys the mic, slower speech for corrections, a pronunciation dictionary for ATC words, and the same voice in Spanish, Hindi and French.
  • ink-whisper word timestamps let the console strike out exactly the words lost in the overlap.
  • Streamed text drives live captions on every screen, about 100 ms behind speech.

Moss: each aircraft's memory.

  • Every plane has its own in-process Moss session holding every clearance it heard, on board.
  • When a call is stepped on, a hybrid search with the garbled fragment the cockpit caught ("Cessna 123, short runway 30") finds the full instruction.
    • The right clearance scores 1.0, and unrelated speech scores 0.0.
  • Metadata filters and sorting give the Clearance agent the aircraft's last tower call in 0.29–0.56 ms.
  • A pilot can ask "Unstep, what was my last clearance?" and get the answer from Moss, spoken by Sonic, in under 1 ms.
  • The memory survives a restart from disk in 21 ms, and each landed flight's log syncs to Moss Cloud.

Elastic: the agent platform, on Serverless.

  • 8 Agent Builder agents:
    • Intercept, Compliance, Ground, Readback Auditor, Audit and Coach, plus two fast variants;
    • 9 tools, each a fixed ES|QL query or index search, so agents only call what they're given.
  • 7 Workflows:
    • agents run in parallel on every stepped-on call;
    • a shift briefing every 8 hours and a nightly review run on schedule.
  • The controller's rulebook (FAA Order JO 7110.65) lives in Elastic:
    • a percolator matches each instruction and readback to its rule (readback matched in 3 ms);
    • hybrid search with Jina reranking finds the paragraph to cite.
  • geo_shape and geo_point:
    • the airport's runway and hold-short areas are shapes, and every aircraft's position is a point;
    • "is the runway clear?" is answered in 0–1 ms;
    • runway incursions are flagged;
    • a Kibana map shows the planes live.
  • ES|QL powers the safety analytics and the plain-English audit trail. Our tools are also exposed over MCP.

The rest of the stack:

  • Hub: a Python/FastAPI WebSocket hub with typed message contracts.
  • ATC console: a 3D digital twin of the airport in three.js, with real taxi routing and readback-first movement.
  • Pilot app: React.
  • Tests: about 600 automated tests, plus a 28-check live end-to-end suite.
  • Demo video: made in Remotion from footage recorded live on the system.

Measured end to end: from the end of a stepped-on call to the pilot hearing the fix took 2.9 s, controller approval included.

Challenges we ran into

  • Our own pronunciation dictionary broke our speech recognition.
    • We respelled "niner" as "NY-ner". Sonic then said something like "N-one-er", and Ink-2 heard "and liner".
    • Runway accuracy dropped to 85%.
    • A text-to-speech → speech-to-text loop over 40 ATC lines caught it, and removing one entry took us back to 100%.
    • Now every voice line we generate goes back through speech recognition before it ships.
  • Agents are slow; radios are fast. An Agent Builder call takes about 5–10 s, but a pilot can't wait that long.
    • The decision card goes out at once from a local check, and the agents' FAA citation is added to the open card when it arrives.
    • Agents can only lower confidence, never change an instruction.
  • Not guessing in a safety system.
    • No model ever writes an instruction: the words come from the controller's own transmission.
    • Hints only ever say stop or wait. Translations must keep every digit.
    • Anything uncertain becomes "say again".
  • Planes that move too eagerly. Our first 3D airport moved aircraft as soon as the tower spoke. We reread FAA JO 7110.65 ¶2-4-3 and AIM 4-3, and now an aircraft moves only after a correct readback.
  • Two teammates, two coding agents, one codebase. Our agents coordinated through Markdown "mail" on a Git branch, with a shell watcher (not a model) checking for replies, and never edited each other's files without asking.

Accomplishments that we're proud of

  • A real, live loop: a stepped-on call is detected, recovered, approved, spoken privately and checked by readback, end to end.
  • Anyone can join: scan a QR, and a judge's phone becomes an aircraft on the frequency.
  • Measured, not claimed. Every number above came from a live run, simulated traffic is labelled, and our known limitations are in the README.

What we learned

  • Voice systems need voice-level tests: text that looks right can sound wrong.
  • In safety-critical AI, the best agent is often the one that refuses to guess, and the human must stay the one who decides.
  • Fast, local memory (Moss) and a slower, smarter agent platform (Elastic) work best together when the slow part never sits on the critical path.

What's next for Unstep

  • False-alarm rates on real recorded ATC audio (ATCO2 and UWB-ATCC datasets).
  • Watching for missing readbacks, stuck mics and conflicting clearances.
  • A Raspberry Pi as a real on-board cockpit computer.
  • Testing with working controllers before any claim about workload.

Built With

Share this project:

Updates

Submission history