When I received the invitation to join the Facebook Hackathon from Tan Duong, we agreed to team up and build a project with Spark AR. That day we got so inspired by a clip with a fairy flying around and then we thought why not make a camera effect that lets people turn into winged creatures. We started Mysterious Wings with the intention of creating an AR world effect that’s optimized for an exciting new product — Instagram Reels. The idea is to let the user choose their favourite wings sets and act, playing with it.
What it does
Mysterious Wings is an augmented reality effect that runs on Instagram cameras. Players will choose wings sets by changing their facial expression, pick the background from their gallery and finish them with an act and poses! The main goal of the effect is focusing on the users’ experiences, helping the users have a fun acting, posing experience, also creating viral insta/facebook stories.
We provide 3 types of wings with corresponding environment for each type. Users can:
- Take video of each wing
- Mode automatically switches wings type when face detection (this effect is great for performing character transitions)
- Touch screen mode to take pictures: the image will be retained, and the user can record video on the background of that image (which helps to make surreal videos)
- Add background photo mode from gallery
How we built it
We discussed and divided the work from the beginning. I do the assets stuffs and Tan Duong does the patches work. Sometimes we switched and helped each other out. We started with the flow of user experience, references of wings and then some concepts ideas. From the 2D concepts and the refs, we build the models, rigged and animated them in Maya. Texture was painted in 3Dcoat and edited in Photoshop.
The whole modeling, texturing process was done back and forth to have the right look and unified in style. Models were set at same scales, rigged with enough bones to deal with and same bones position so our teammates won’t get frustrated. Animations in each wings set was splitted in 2 clips (start and idle) exported into 1 FBX file.
Later on, they were all assembled and tested in Spark AR. Bugs, missing points were adjusted in this step. Next, we test and adjust the materials to fit the environment. After that we add a few particles to fit the style of each wings set (Sparks with demo, Feathers with Angel, Sparkle with Fairy). Last step was working on the patch system to get them changed when people change their expression. Add the ability to browse the gallery and pick background. Then add the new function “Reels” that give users the ability to turn into winged creatures and fly away.
We made certain steps that the user will follow
- Stand in front of the camera and posing, act.
- Hit play button to freeze and change act.
- Using facial expression to change wings sets.
- Browse the gallery and choose background.
2.Build the effect
The work schedule is updated every day.
For Design: We spent a lot of time to calculate the number of models, bones, materials and textures so the file size is not too large, how to compress the texture to ensure the best image quality with smallest file size.
For Dev: We start with add and aligning the object layers to work well in all circumstances. Add texture and texture to look exactly what was being built with 3D software. Make changes to the respective wings, head, and environment, using the native ui picker to do the transformation. Add interactions as users use. Adjust lighting and colors for the final time before submitting.
When the patch was completed, we once again assembled them and tested the accuracy of the logic We have tested many times. The process took us lots of time, but fortunately there were not too many issues and we were still in good control
Challenges we ran into
Building visuals and users’ experience, user-centered to creativity and act, posing. Working time: We are full-time employees for the company, we are afraid of the deadline because it seems that both Tan Duong and I are very busy with our own work.
- For Design: We have to adjust a lot so the assets unified in style and fit the real world
- For Dev: The way of assembling the wings on the human body is a bit difficult because the camera cannot capture the face at a long distance. The combination of background and freeze modes is imported from the gallery.
Accomplishments that we're proud of
- Our first project was completed despite many difficulties.
- Learned how to fit the objects with whole body.
- We were very busy, but still on schedule.
- Exceeding the production plan is not only a filter for Facebook but also for Instagram.
- Making all the functionality work properly without sacrificing any.
What we learned
- In the process of making filters, we have been learning many useful lessons. Fortunately, we agreed on solving the root problem including ideas, scripts, and how to proceed.
- Learn how to plan and work remotely with a filter project.
- Need the most detailed brief and script (including logic and time spent on animation) to be able to understand the project most clearly.
- Not surrender, cause we got so little times to worked on this (knowing the deadline was too late and I still not get any good result) and almost quit the project but then we choose sleepless night and finish it.
What's next for Mysterious Wings
We will continue to update some new contents for filter, such as:
- More wings sets to choose from.
- Ways to customized it.