Taxiway Reliability

A segmentation model for aircraft taxiing — plus a layer that tells you, with no answer key, when not to believe it.

Comment

Taxiway perception that knows when it's wrong

A segmentation model for aircraft taxiing — plus a layer that tells you, with no answer key, when not to believe it.

Inspiration

A runway incursion is one of the few aviation failure modes that still kills people regularly. The fix everyone reaches for is computer vision: put a camera on the aircraft, segment the taxiway, feed it to a localization filter.

We started by reproducing a 2026 Stanford paper that does exactly that: Prince et al., A Filtering Framework for Aircraft Localization During Taxiing Using Semantic Segmentation. And once we had it working, the thing that bothered us wasn't the accuracy. It was that when the model is wrong, it is wrong silently. It returns a clean, confident-looking mask whether or not it has any idea what it's looking at. Snow on the shoulder, standing water, a low sun; the mask looks exactly as trustworthy as it does on a clear dry taxiway.

A filter downstream has no way to tell those apart. That's the actual safety problem, and nobody in the paper addresses it.

What it does

Two things.

1. It reproduces the paper's segmentation stage. DINO ResNet-50 encoder, U-Net decoder, per-channel multilabel Dice loss with inverse-frequency class weighting, trained at full 1080×1920 on all 10,000 images of the Stanford VisualTaxiULI dataset.

model hold-short centerline pavement macro IoU
ours 0.7041 0.7102 0.8803 0.7649
authors' published weights 0.3505 0.7275 0.9824 0.6868
paper, Table 3 0.6645 0.7317 0.9870 0.7944

2. It adds a per-frame trust score the paper does not have.

Here's the idea. The network emits a probability per pixel. A mask that commits has nearly every pixel pinned near 0 or 1. A mask that's guessing hedges, and the hedged pixels are exactly the ones sitting on a boundary the network can't actually place. So we measure the fraction of each predicted region falling in the undecided band between sigmoid 0.05 and 0.95.

That's it. Just one more threshold comparison on tensors we already have.

On 500 held-out frames, calibrated on a disjoint 500:
Spearman ρ, trust vs. actual frame IoU +0.42
AUC, flagging the worst 10% of frames 0.76
mean IoU, 3 most-trusted frames 0.84
mean IoU, 3 least-trusted frames 0.55

And then you can act on it. A localization filter doesn't have to use every frame. If it abstains on the ones the layer distrusts, the frames it does act on get measurably cleaner:

coverage macro IoU Δ
100% 0.7302 —
80% 0.7376 +0.0075
70% 0.7440 +0.0139
50% 0.8112 +0.0810

Half the frames carry nearly all of the error. A filter that knows which half to ignore gets a much better signal than one that averages everything together.

Look at the contact sheet in the repo: top rows are frames the layer trusts, bottom rows are frames it flags, and the error panels show red where the model hallucinated pavement and yellow where it missed it. The flagged frames are the snow, the puddles, the washed-out shoulders. Nobody told the layer which was which.

How we built it

On a GPU that isn't supposed to do this. The codebase started on a Mac using Apple's MPS backend at quarter resolution. We ported it to an Intel Arc B580 — not CUDA, not ROCm, Intel's XPU backend through Level Zero. Almost nothing in the ML ecosystem assumes that card exists.

Getting 1080×1920 training into 12 GB took:

  • bf16 autocast, with the Dice loss forced back to fp32 (it sums ~2M pixels)
  • gradient checkpointing through the whole encoder
  • a micro-batch autotuner that measures peak memory at 1 and 2 samples and extrapolates the rest arithmetically

That last one matters more than it sounds. This card does not raise a clean out-of-memory error; it raises UR_RESULT_ERROR_DEVICE_LOST, which survives process exit and requires a reboot. We learned that the hard way: an allocate-until-it-fails probe wedged the GPU mid-project and, because the B580 also drives the display, artifacted the screen until we rebooted. The autotuner now refuses to allocate anything it can't prove fits.

Final training run: 16 epochs, ~11 hours, peaked at 5.7 GB of 12 GB, no artifacting.

Challenges we ran into

The paper doesn't reproduce from its own published weights. We downloaded the authors' checkpoint and evaluated it. It recovers centerline to within 0.004 and pavement to within 0.005 of Table 3, which is strong evidence our metric is defined the same way theirs is, but scores 0.3505 on hold-short against a reported 0.6645.

Digging into their training code, it saves two checkpoints: best-average-IoU and best-worst-class-IoU. Hold-short is always the worst class, so the second one is precisely the one that maximizes it. Only the first was published. Table 3 looks like it draws on both. We can't confirm that, but we can show the published weights don't produce the published table.

The stated hyperparameters aren't what the code runs. Table 2 says batch 8, 2 epochs. The released code defaults to batch 4, 50 epochs, and the documented full-training command overrides neither. Two epochs measurably undertrains: 0.678 macro on the full dataset, worse than 4 epochs gets on half of it.

9,972 broken symlinks. The dataset split shipped as absolute macOS paths.

Accomplishments we're proud of

Honestly? The negative results, and that we kept them in.

We predicted per-class threshold tuning would help, and it didn't. The loss is inverse-frequency weighted, which hands pavement ~0.2% of the loss weight against hold-short's ~87%. We reasoned pavement was therefore trained to an over-conservative boundary and would want a lower decision threshold.

Wrong. The optimal pavement threshold is 0.75, higher, not lower, and tuning all three classes buys +0.0023 macro IoU, which is noise. Pavement's weakness is not a thresholding artifact. The hypothesis was clean, testable, and false, and it's in the README.

Our first trust signal was the weak one. We started with test-time augmentation: predict on the frame, predict on its mirror, measure disagreement. It works; ρ +0.30, AUC 0.57, but it's mediocre and costs a second forward pass. The boundary-ambiguity signal beat it on every metric and is free. We selected between them on the calibration half, not the half we report on.

Every number in this project is calibrated on one half of the official validation split and reported on the other.

What we learned

  • An uncertainty signal doesn't have to be exotic. We reached for MC-dropout and TTA first; the thing that actually worked was already sitting in the logits.
  • "The model is confident" and "the model is right" are different claims, and almost no vision project checks the gap between them.
  • Reproducing a paper teaches you more than reading it. We found two discrepancies that only show up if you actually run the released code against the released weights.
  • Consumer GPUs outside the CUDA monoculture are genuinely hostile territory, and a memory bug can cost you your display.

What's next

  • Validate the trust score against downstream localization error, not just per-frame IoU. That needs the paper's UKF/PCA filter stage and a 193 GB trajectory archive we didn't have room to download.
  • Feed the trust score to the filter as a measurement covariance instead of a hard abstain — a soft weight is strictly more information than a threshold.
  • Pin a random seed. Currently runs aren't bit-reproducible.

Built With

Share this project:

Updates