What if we made a room where one giant wall was a window into an alternate dimension?

The cosmos teems with a multitude of worlds, as well as myriad dimensions of reality. These realms, known as the Planes of Existence, hold elemental energy in its purest form.

It would be cool enough to witness one with your own eyes – but we thought, what if you could manipulate the energy inside one?

Earthbound is an interactive art installation that towers several feet above you, responds to your every move, and simulates what it might be like to gaze into another world. The vastness of it is meant to make you feel like you're part of something bigger: that this world is more than what we can see or what we know, and yet we're not powerless in it.

Enter the simulation.

Our Processing sketch detects motion in your webcam by comparing every video frame to the one before it. You can dance, jump, wave your arms, anything, and all of those motions will push around the 24,000+ particles floating on-screen. Whenever one particle collides with another (or the edge of the canvas), it glows bright green before slowly fading out to black. And the colors of each particle are blended with pixels from a photo – so over time you should see a forest materialize :)

We brought this into Unreal Engine in order to render our concept and show the scale of it. In Unreal, we created a scene with walls, reflective floors, dimmed lights, a video of the Processing sketch (embedded in an unlit emissive material applied to a plane), and a mannequin for reference. We then added a post-processing volume and enabled bloom to make the plane glow like an LED screen.

Our ambient music was generated in Orca!

Challenges we ran into

  • We actually tried to make this in openFrameworks first, but the last release of ofxFlowTools isn't compatible with the most recent release of OF. (And up-to-date Macs can't run the archived version of OF that it is compatible with, because it's 32-bit. Sucks.)
  • We were completely new to ORCA, so we had to spend quite some time learning the basic operators before we were able to produce our own music.
  • Our computer crashed when the music was halfway done... Fortunately we still had about 4 hours left until the deadline.
  • Then, 2 hours before the deadline (and right before hitting Record!!), Unreal crashed while compiling shaders and the project files got corrupted. We had to build the scene and perfect the post-processing values all over again. We had a sneaking suspicion that the mannequin's shaders were what slowed Unreal to a crawl, so we left him out of the new project (and thus the video).

Accomplishments that we're proud of

  • Coding collaboratively in Replit! Typing at the same time and seeing each other's cursors/highlights was so much easier than Git commits.
  • ORCA's programming language is rather strict, but we played around with it to form a functioning & musical H in the end — H for Hackathon!
  • Overcoming time zones. We were 12 hours apart!

What's next?

  • Master ORCA...?
  • It would be cool to see this in person some day :')


PixelFlow: a Processing/Java library for high-performance GPU computing. Features fluid simulation, flow field particles, softbody dynamics and more.

HundredRabbits: Thank you for making ORCA so beginner-friendly!

Thank you Richard Whaling for your programmatic music workshop at UH22!!

Inspired by Refik Anadol's larger-than-life installations... (Machine Hallucinations: ISS Dreams)

...and the 5th Edition Reference Doc for Dungeons & Dragons - specifically the Planes of Existence chapter.

Built With

Share this project: