Butterfly Swarm

Inspiration

The chaotic nature of butterflies has always amazed me; seeing how many creatures interact with each other without any central coordination is mesmerizing to me.

We are now living through times when going outside isn't possible for many of us so, I decided to recreate this as an Augmented Reality experience.

What it does

It allows users to experience being surrounded by butterflies or, see many of these flying around the scene with the use of the plane tracker, depending on the camera that's used. This duality is important for Reels as it can function correctly and be used with either camera giving more flexibility to the user.

How I built it

I started by researching about emergent behavior - complex behavior arising from the relationships between the individual parts of a system - and watching butterfly footage. Eventually, I decided to use Craig Reynolds' boids algorithm to simulate emergent behavior.

The algorithm is mostly controlled by three simple steering behaviors:

Alignment: Individuals will try to align themselves with the center of the local flockmates.
Separation: Individuals will steer away from nearby local flockmates.
Cohesion: Individuals will steer toward the average position of local flockmates.

With the keyword being local, as in nature creatures don't have an accurate and complete picture of their environment, but rather have a limited perception of it.

After implementing the algorithm I focused on the 3D model that acts as the "body" of the entities. I built, rigged, and animated the model in Blender. To reduce the size of the project, there's only a single color texture that get's slightly randomized for every butterfly.

Challenges I ran into

Efficiency

The algorithm has a Big O notation of O(n)² which means that the number of operations grows exponentially with the number of boids, not only that but, these operations get run every frame - 33 milliseconds!

To improve efficiency and add some useful functions, I created my own Vector class that handles all the vector operations and was easy to modify and adjust to the project's needs.

All the functions were thoroughly tested and most can be run up to 25.000 times per frame or more.

Keeping the butterflies from flying away

Most implementations and simulations that use Craig Reynolds' algorithm take place in a 2D or 3D environment with solid bounds from which the boids can bounce or wrap around but, in an AR experience walls would be a huge distraction (and look horrible) so instead, I created an "invisible force field" bound to a user's face that constrains the butterflies' movement. In the front camera, it also has a lower distance limit that prevents the 3D objects from crossing right through the middle of the person segmentation mask which would also look bad and take a user away from the experience.

Patterns and cycles

Even though randomness is involved in many parts of the system over time some patterns start to emerge, distracting from the experience. To solve this the parameters had to be tweaked until no cyclic behavior eventually appeared in the following scenarios:

A static attractor (face).
An attractor (face) that moves back and forth in an oscillating manner.
An attractor (face) that quickly changes positions - for example, if the face of another person starts being tracked.

Difference in computing power across devices

Across brands and generations phones' CPU power changes significantly, trying to come up with a fixed number of boids that all devices in the market can handle would be impossible so instead, a "lag monitor" was implemented that monitors the user's experience. If there is on average more than 40ms between frames - 25fps - the number of boids is reduced and the ones that aren't part of the calculations are hidden.

Accomplishments that I'm proud of

How light the filter is, it's is less than 200kb so it can quickly be downloaded even on a slow wifi network.
How much I've learned in the process, I gained a deeper understanding of Javascript and learned a lot about 3D objects which I previously had zero experience with.
All the challenges I overcame.

What's next for Butterfly Swarm

As I gather feedback from the community and people who try the filter I may make small revisions to perfect the algorithm's parameters, the model or, the textures. Also, a lot of the knowledge gained during the making of this effect will make its way into effects I have in the making.