SKYWARD

GIF
Weather & Wind visualiser
GIF
Pilot webcam & script for training
GIF
Airplane live turbulence simulator
GIF
Record button for audio input
GIF
ShapesXR scene design
Initial iPad sketch
Serious runway incursions, by cause
Additional data on runway incursions
Total runway incursions (FAA)
GIF
ShapesXR export of scene

Inspiration & Insights

“In a review of FAA reports and a NASA database, the Times found there were at least 46 close calls involving commercial flights in July (2023) and runway incursions are 25 percent higher than a decade ago.” - PBS (1)

Safety in the skies is a topic on everyone's mind. For most of us here coming from the 4 corners of the world, flights are our way to conduct business, see our close ones, and connect anywhere on earth. With recent flight incidents seemingly appearing more frequently in the news, we felt inspired to find a practical solution that could have an impact on us all.

Our focus naturally gravitated towards the importance of Air Traffic Controllers, being able to coordinate hundreds of landings and takeoffs in a day, with sometimes multiple happening in the same minute.

During our research into ATC (Air Traffic Control) safety, 3 points stuck out.

Serious runway incidents are on the rise in the last decade, and more of these incidents are coming from air traffic controller mistakes. (2)
Air traffic controllers are often using two-dimensional visualization methods that lack a clear way to observe altitude and longitude/latitude information. (3,4)
Due to a shortage in the recruitment of air traffic controllers, current trained professionals are often pushed into "10-hour shifts 6 days a week", per the NYT (5)

Considering all of these factors, we believe there are multiple ways to solve the safety issue around airport runways. As many reports claim the Air Traffic Controller shortage could last up into the 2030s (6), technology might offer a reliable solution to make our skies safer.

Our solution

SKYWARD uses a combination of Mixed Reality features to add a layer of dynamic information over the traditional tools used by air traffic controllers. To make the experience seamless, all interactions are controlled by hand and voice tracking.

As most tools used to visualize airplanes use a 2D top view or a 3D view on a 2D screen, it can be hard to clearly distinct between flights at different altitudes. With SKYWARD, a 3D representation of any airport with surrounding and incoming flights is transported into your environment and can be understood at a quick gaze. Additional layers of information can be added with the simple push of a button: surrounding airplanes, wind patterns, and advanced storm patterns.

Our goals

Improve air traffic safety at every airport and in the skies
Increase the information quality visible to air traffic controllers by providing a more accurate way to visualize flight data
Reduce the reliance on 2D screens and hopefully improve the quality of work of air traffic controllers

How we built it

As our focus was on having a hands/controller-free approach, we used a combination of tools to turn our idea into an (augmented) reality.

We decided to develop our project on the Meta Quest 3, using its advanced mixed reality pass-through capabilities. Meta's all-in-one SDK on Unity allowed us to iterate together on different features such as hand tracking and voice recognition. Voice recognition is handled by Wit.AI.

For our 3D assets, we designed our UI/UX in ShapesXR, allowing us to port our models directly into Unity. Other assets such as the airport and airplanes were found on Sketchfab/CGTrader. (7,8,9)

All our progress in Unity was synchronized using Codeberg with Fork, allowing us to easily work in parallel on multiple features.

Our 3D airplane animations were created in Blender, where multiple animation tracks can be executed on the same model.

Challenges we ran into

One of our challenges came with the use of the Meta Voice SDK with Wit.AI. We implemented Meta Presence Platform's Voice SDK and trained custom voice detection models, but ran into some challenges with the accuracy of audio detection within Unity. We also ran into a few challenges setting up our Codeberg project and synchronizing large project files.

Accomplishments that we're proud of

Animated weather shader with wind and storm patterns
3D airplane animations
State manager for different animation triggers
Visuals and simplicity of different toggles

What we learned

One of our biggest learnings was the importance of Air Traffic Controllers. The role they play at airports around the world cannot be understated, and we were not aware of how deeply the shortage crisis has been affecting the industry.

We also greatly broadened our technical horizons by learning to integrate several Meta Presence Platform technologies, cinematic video footage, custom shaders, and audio, into an immersive MR experience. We used ShapesXR for the first time, and learned how to import 3D models into a Unity scene. We also improved in creating 3D models and animations in Blender.

What's next for SKYWARD

Real-time information streaming coming from various instruments such as ADSB, pilot audio communications, etc. to use for real-world scenarios.
Create new training scenarios for situations varying in complexity.
Additional visualization methods for different weather and sky conditions.
Introduce the possibility of collaboration for air traffic controllers through a shared experience (multiplayer).
More robust voice recognition system.
Conduct research on how effective our solution would be for real-world scenarios or for training.

Thank you for reading! :)