Sky-C2

AR application that allows plane enthusiasts and air traffic control (ATC) workers to instantly identify aircrafts through Sky-C2's HUD. Aircraft data is generated using OSINT.

Comment 2

Inspiration

In an airport control tower, identifying the aircraft outside the window is still surprisingly manual. Controllers look out the window, often with binoculars, then look down at a radar display, then cross-reference paper flight strips and lookup tables of callsigns and aircraft types to work out which dot is which plane. All of that happens in a job where every second of attention counts.

Watching the constant traffic over Georgia Tech into Atlanta, the world's busiest airport, we asked: what if the information came to the controller's eyes instead of the other way around? Look at an aircraft and immediately see who it is, where it's going, and whether something is wrong, all in one view. That's Sky C2: a command-and-control view of the sky.

What it does

Sky C2 consolidates the out-the-window view, the radar picture and the flight information into a single heads-up display. The prototype runs on a phone in a $10 Google Cardboard.

  • Visual identification without binoculars or lookups. A compass "headband" places every aircraft at its real direction, with plane and helicopter icons. Center one and a card shows its callsign, airline, aircraft type, route, altitude, climb/descent rate, speed and distance.
  • Situational awareness at a glance. Aircraft are color-coded: blinking red for emergencies (including squawk 7500/7600/7700), purple for medical flights, green for military, grey on the ground. A toggle hides ground traffic.
  • Radar built into the view. A heading-up radar mini-map shows the whole airspace around you and rotates as you turn your head, so you don't need to look down at a separate screen.
  • Hands-free queries. Say "Hey Grok" and ask about the aircraft you're looking at ("Where's that flight headed?", "Is that one climbing?"). Grok answers out loud using live data for that exact aircraft.
  • A shared picture. A live view streams what the wearer sees, overlay included, to another screen, so a supervisor or teammate sees exactly what the controller sees.

How we built it

We didn't use any computer vision. It's all sensors and geometry + OSINT. The headset knows where it is (GPS) and which way it's facing (compass plus gyroscope), and live ADS-B data says where every aircraft is. We flatten the problem to 2D. For each aircraft we compute the bearing from the viewer:

$$\theta = \operatorname{atan2}\big(\sin\Delta\lambda\cos\varphi_2,\ \cos\varphi_1\sin\varphi_2 - \sin\varphi_1\cos\varphi_2\cos\Delta\lambda\big)$$

and the signed angle between that bearing and where the viewer is looking, $h$:

$$\delta = \big((\theta - h + 540) \bmod 360\big) - 180$$

An aircraft is in view when $|\delta| \le \tfrac{\text{FOV}}{2}$ and "in focus" when $|\delta| \le 7.5^\circ$, a 15° cone. Its horizontal position comes from a pinhole-camera projection, so icons line up with the camera image:

$$x = \frac{W}{2}\left(1 + \frac{\tan\delta}{\tan(\text{FOV}/2)}\right)$$

The stack:

  • Web app (Vite + TypeScript): stereo camera view, orientation math, and a canvas HUD drawn identically in both eyes.
  • Flight service (Python/FastAPI): pulls live aircraft from adsb.lol every second, normalizes them, computes bearings and distances, adds routes, and flags helicopters, emergencies, military and medical flights.
  • Voice service (FastAPI + xAI): builds a prompt from the focused aircraft and asks Grok (grok-4.3, no reasoning, about 0.9 s per answer). The answer is spoken with Grok's text-to-speech voice, and wake-word detection runs in the browser.
  • Live view: WebRTC streams the composited view peer-to-peer to a laptop, with one-click recording.

Challenges we ran into

  • "Heading" isn't where the camera points. The phone's compass reports the direction of its top edge, but in a headset the phone is sideways and we care about the rear camera. We compute the camera's direction from the full device rotation instead: $\mathbf{c} = -R\,\hat{z}$, so heading $= \operatorname{atan2}(c_E, c_N)$. We also correct for Atlanta's roughly 5° magnetic declination.
  • Cardboard is unforgiving. A phone sitting slightly crooked in the headset gave us double vision, and cropping a 16:9 camera stream made everything look zoomed in. We added in-headset calibration (tilt, lens spacing, shift, zoom, field of view), controlled by taps and locked by default so a brushing cheek doesn't change anything.
  • Voice cut people off. Chrome's speech recognition decides you've finished at the first short pause. We switched to waiting for 1.3 s of silence before sending the question.
  • Rate limits. At 1-second updates, adsb.lol started rejecting requests (HTTP 429). GPS jitter made it worse by bypassing our cache. We snap positions to about a 1 km grid, back off when rate-limited, and always fall back to the last good data instead of showing an empty sky.
  • Getting the data right. Helicopters without a type code were described as "slow-moving planes", and a "lifeguard" (medical priority) status looked like an emergency. We derive aircraft kind and status on the server and pass them to both the HUD and the AI.

What we learned

  • Geometry and sensors can do a lot of what you'd assume needs computer vision.
  • In a heads-up display, less is more: controllers need the right information in their line of sight, not more screens.
  • Clear contracts between teammates (and their agents) let three people build in parallel with almost no merge conflicts.
  • Always plan for the demo environment: an indoor demo mode, fallbacks for every API, and a live view so others can see inside the headset.

What's next

Sky C2 is a prototype on consumer hardware. Phone compasses drift, especially indoors, and public ADS-B feeds aren't certified for operational use. Next steps would be a proper AR headset with precise head tracking, a surveyed tower position, the airport's own surveillance feeds, and placing labels at each aircraft's true height in the sky, not just its direction.

Built With

Share this project:

Updates

Submission history