Flight Loom

Flight Loom transforms the visible motion and colors of drone video into a living digital textile with a reactive soundscape—an interactive artwork analyzed and remixed locally in your browser.

Comment

Inspiration

A drone flight is normally watched as a finished rectangle, but every turn, lift, shadow, and change of color carries its own rhythm. Flight Loom began with a question: What if a flight could leave behind a textile woven from what the camera saw?

The goal was to turn an ephemeral path through the air into participatory digital craft—something viewers could watch, hear, and rearrange rather than simply replay.

What it does

Flight Loom connects three layers: the source video, an expressive motion sample, and a living digital textile.

The bundled demo follows a 32.5-second recorded flight divided into five movements. As it plays, the current movement drives a woven Canvas artwork and a generative Web Audio soundscape. Visible frame change affects thread density and variation, brightness-center drift bends the weave, and sampled colors become its palette.

Viewers can select any woven band and:

  • Move it earlier or later
  • Reverse its direction
  • Repeat it up to three times
  • Return to the original source timing

These edits change the artwork, not the source video.

Viewers can also choose their own MP4, MOV, or WebM clip. Flight Loom samples 24–32 low-resolution frames directly in the browser, creates up to five movements, and keeps the selected video on the device instead of uploading it to an application server.

How we built it

Flight Loom is a React and TypeScript web experience built with Next.js.

The browser analyzer creates a temporary local object URL, decodes selected frames with HTML5 Video, and draws them to a small off-screen Canvas. It derives average RGB color, frame-to-frame luminance change, and luminance-centroid drift. These are deliberately lightweight artistic signals—not optical flow, GPS data, or aircraft telemetry.

Each movement becomes a deterministic FlightSegment containing duration, energy, horizontal drift, vertical drift, scene change, palette, repeat count, and direction. A Canvas 2D renderer translates that data into layered threads, curved bands, color washes, and a playback shuttle.

The soundscape uses the Web Audio API. Energy influences note timing, lift changes pitch, horizontal drift changes stereo position, and scene change shapes filtered wind, tone color, and reverb. Because browsers restrict Web Audio startup, the sound system initializes from the viewer’s trusted play gesture while retaining a default-on preference and manual off control.

The recorded demo was prepared as a browser-friendly 720p H.264 MP4 with a VP9/WebM fallback and a WebP poster. A 15-test regression suite covers the movement model, audio mappings, media assets, rendered experience, and native-video sound unlock behavior.

Challenges we ran into

The first challenge was finding a visual mapping that felt connected to the footage without pretending to measure more than it does. Frame difference and brightness-center drift can create expressive motion cues, but they are not precise camera tracking. The interface and project language therefore make that boundary explicit.

The second challenge was maintaining two related timelines: the original source timing and the viewer’s remixed band order. Separating source movements from editable artwork state allowed playback metrics to remain synchronized even after a band was moved, reversed, or repeated.

Browser media behavior was another challenge. Native video controls do not always expose their gesture soon enough for Web Audio initialization. Capturing pointer and keyboard gestures at the video container fixed silent first playback while preserving browser autoplay rules.

Finally, the original 4K HEVC footage was too large and inconsistently supported for a web demo. Creating compact H.264 and VP9 versions made the experience practical across more browsers without modifying the original source footage.

Accomplishments that we're proud of

  • Turned real recorded drone footage into a synchronized visual and sonic artwork
  • Built custom-video analysis that runs locally without an application-server upload
  • Kept source playback, motion metrics, woven output, and soundscape connected
  • Made the result genuinely interactive through reordering, reversing, and repeating
  • Clearly separated artistic visual approximation from telemetry or scientific measurement
  • Created automated coverage for the data, audio, media, rendering, and playback paths
  • Shipped a working public browser demo and public source repository

What we learned

Simple signals can become expressive when their mapping is coherent and visible to the audience. The project did not need a heavyweight recognition model; average color, frame change, and centroid drift were enough to support an understandable artistic system.

We also learned that generative work benefits from explanation. Showing the source video, sampled values, and resulting textile together makes the artwork feel participatory rather than mysterious.

On the engineering side, media playback and Web Audio are tightly constrained state machines. Gesture timing, seeking, cleanup, and the separation of source state from remix state all mattered as much as the visual renderer.

What's next for Flight Loom

  • Exporting high-resolution textile stills or short audiovisual renders
  • Adding adjustable mapping and sound-intensity controls
  • Supporting longer clips with more adaptive movement boundaries
  • Saving local remix recipes without uploading the original footage
  • Expanding mobile-browser and accessibility testing
  • Exploring richer motion analysis while keeping the result explicitly artistic rather than presenting it as telemetry

Attribution

The bundled demo is a derived excerpt from project-provided drone footage, included with the creator’s authorization. The original footage predates the hackathon; all Flight Loom analysis, interaction, generative artwork, soundscape, and web implementation were created during Hack the Arts 2026. The project uses open-source React, Next.js, TypeScript, Vite/vinext, and standard Web platform APIs.

Built With

Share this project:

Updates