Sky Lantern - make a wish

Inspiration

Year 2020 was off for a good start. Following half a year long social unrest in Hong Kong is the coronavirus outbreaks. Political and international conflicts escalate one after another. Natural disaster is unstoppable burning down million acres of lands and killing billions of animals. Can humanity survive another century? I don't know.

In asian and elsewhere around the world, Sky Lantern is a traditional culture. The lanterns are like small hot air balloons made of paper. People wrote down their wishes and sent them to the sky hoping for better luck. Unfortunately, this tradition has been made illegal in many countries due to potential fire tragedies.

Luckily with augmented reality technology, we can bring this back to live. Traditions and cultures play an important role in humanity. I believe good technology is about respecting cultures and bringing them new forms to fit modern world instead of trashing them away completely. Technology should also connect people instead of tearing them apart. It would be nice to create a picture together with people around the world, so let's do this in a live social sharing platform!

What it does

Facebook live streaming allows audiences to submit comments and reactions in realtime. In this AR effect, the host user will start the effect with live streaming. Audience users then comments and send reactions. The effect will read the content in real time, paste them in the sky lanterns and send them up to the sky. The experience is exactly like traditional culture, but in a virtual format.

How I built it

First of all, we built a sky lantern 3D model with carefully crafted texture map, because we need to create a texture dynamically later on. Next step is to dynamically generate texture using users content. Basically, it means blending in two textures (the base lantern texture and a new texture generated from users).

After succeeding in generating one lantern, we then wrote a module to spawn the lantern periodically because we want to AR effect to show many lanterns instead of just one. Finally we animate them to create a coherent effect.

Challenges I ran into

The greatest challenge of this project is to blend-in the users content into the pre-built sky lantern. Life would be much easier if SparkAR can support uploading user image as texture. Or allowing us to create texture programmatically by manipulating pixels. However, it seems to me that those functionalities are not available. Finally, we resolve to the hard way by hacking the shader modules. We generate a font texture containing all the characters and numbers, and then write a shader program to dynamically blend them with the lantern texture in the shader level.

Accomplishments that I'm proud of

This is the first time I kind of writing a shader from scratch. Being a developers working with high-level programming languages, this is an interesting experiences, and I'm happy with the result.

What I learned

Putting technical aside, I learnt more about the traditions and culture related to sky lanterns. It also helps me rethink different aspects of technology. I wish my future projects can also be more connected with people.

What's next for Sky Lantern - make a wish

Right now, it's only supporting English words. The immediate next step is to make it multilingual as well as supporting emojis. Later on, it would be nice to be able to take in arbitrary images from users. Technically speaking, using uploaded images as texture directly is easier, but we would have to wait for facebook team to support that.