Icarus

Icarus is a pair of wearable wings that translate movement into game controls using rotary encoders and camera tracking, powering Cloudbounded, Skyborne, and other games through our SDK.

Comment

Inspiration

Inspired by the theme Into the Skies, we wanted to build a more interactive way to play games. The Wii Remote and VR/AR headsets let players participate through physical movement, and we wanted to explore that interaction through wearable wings.

That became Icarus, a physical controller that translates wing movement into game inputs. We built Cloudbounded and Skyborne to demonstrate the projects adabtability to different games, and developed an SDK and input adapters so the controller can extend to other games (such as GeoFS! )

What it does

Icarus combines wearable wings, rotary encoders, and camera tracking. The encoders measure wing rotation, while the camera tracks colored markers to estimate wing position and movement.

In Cloudbounded, players flap to climb, hold their wings spread to glide, and lower and tuck their wings to dive. Players can also capture their face and use it as the game’s character.

In FlightBorne, players simulate a bird by flapping, diving, and gliding in their wings.

For other games, our controller software maps movement to configurable keyboard inputs. An optional virtual gamepad output exposes movement as analog joystick input.

How we built it

Wearable hardware

The wing assembly gives players a physical interface they can move with their bodies. Rotary encoders measure rotation at the wing joints, providing a direct measurement of wing movement. Camera tracking provides a separate view of the wings’ positions.

Camera tracking

We built the tracking pipeline in Python using OpenCV and NumPy. Rather than estimating the entire body pose, it detects colored markers attached to the wings and body.

Each camera frame passes through several steps:

  1. We convert the image from BGR to hue, saturation, and value so we can isolate the marker color.
  2. We threshold the color by having Configurable tolerances determine which pixels match. Hue comparisons handle the wraparound at the ends of the hue range.
  3. To remove noise, a morphological opening operation removes small isolated pixels.
  4. We find marker centers by extracting contours, rejecting blobs below a minimum area, and calculating each remaining blob’s centroid using image moments.
  5. Then, we calculate movement by having marker coordinates become wing heights, distances from the center, and wingspan measurements.

Cloudbounded uses three markers: one on each wing and one on the body. The bridge selects the three largest matching blobs, ignoring smaller distractions, and sorts them horizontally to identify the center and wings.

Wing heights are measured relative to the body marker. This helps distinguish wing movement from the player moving their whole body up or down.

Calibration and gesture recognition

Before playing, the player holds their wings in a comfortable spread position and calibrates. We store the neutral wing heights and wingspan as a reference.

The game interprets movement over time rather than treating every frame as a separate gesture:

  • Flap: Both wings must show significant vertical movement and a direction reversal within a 700-millisecond window.
  • Glide: When flapping stops, the controller returns to glide.
  • Dive: Both wings must fall below their calibrated heights, and the wingspan must narrow below 65% of the calibrated span, held for 350 milliseconds.

Small movements and one-wing motion are filtered out of flap detection. Hysteresis prevents rapid switching near gesture thresholds, and active flapping takes priority over a momentary tuck. Losing tracking clears the movement history so old measurements cannot trigger a new gesture.

Connecting tracking to Cloudbounded

We built Cloudbounded in Godot 4 with GDScript. A local FastAPI bridge connects the Python camera pipeline to the game.

Camera processing runs in a background thread, while Godot polls tracking state independently of rendering. The bridge provides endpoints for starting the camera, retrieving a preview, calibrating the wings, capturing the player’s face, and reading the current flight state.

The game converts flap, glide, and dive states into different target vertical velocities. Acceleration moves the character toward those velocities gradually, making transitions smoother than instantly changing speed.

If tracking becomes invalid or stale, the game pauses the run. The player can recover tracking and resume or switch to keyboard controls.

Bringing the player into the game

Our face-capture pipeline uses MediaPipe face landmarks and portrait segmentation to create a transparent character image.

It detects the largest face, crops around the head, and combines segmentation with a landmark-based mask to remove the background, clothing, and neck. The resulting transparent image becomes the player’s character in Cloudbounded.

Camera processing and face extraction run locally. The game keeps camera frames and captured portraits in memory.

SDK and other-game support

Our integration layer exposes movement data through an HTTP API, allowing games to consume tracking measurements without implementing the computer vision pipeline themselves.

We also built an input adapter for existing games. It calculates:

  • Roll from the angle between wing markers.
  • Pitch from wing height relative to the calibrated neutral pose.
  • Flap events from a sharp downward movement over a short time window.

The adapter normalizes movement, applies a dead zone, and smooths the output to reduce jitter. Keyboard output uses proportional key pulses: small movements produce intermittent presses, while full movement holds the key continuously.

Key mappings are configurable, and an optional virtual Xbox controller output maps movement to an analog stick. When markers are lost, the adapter releases held inputs to prevent stuck controls.

Challenges we ran into

The main challenge was turning noisy physical movement into intentional game controls. Lighting changes affect color detection, markers can become obscured, and moving the whole body can resemble moving the wings.

We addressed these problems with adjustable color thresholds, minimum blob sizes, body-relative measurements, calibration, and gesture detection over time. We also needed tracking and rendering to operate independently so camera processing would not block the game.

Accomplishments that we’re proud of

We built a wearable controller, a camera tracking pipeline, a custom Godot game, and an integration layer for other games. We also added calibration, live tracking feedback, configurable inputs, and a face-capture system that puts the player directly into Cloudbounded.

What we learned

Physical controllers require more than accurate measurements. The software needs to distinguish deliberate gestures from jitter, accommodate different players, and respond predictably when tracking fails.

Building Icarus taught us how hardware, computer vision, input processing, and game design influence one another.

What’s next for Icarus

We want to improve the wings’ comfort and durability, refine tracking across different lighting conditions, and expand game compatibility. We also want to develop clearer SDK documentation and example integrations so developers can build their own experiences around Icarus.

Built With

Share this project:

Updates

Submission history