We were inspired to make a filter that can help people make interesting choreographed dance videos utilizing an on-screen portal that dramatically alters the users appearance when interacted with. Many users love to make videos that make use of dramatic transitions and showcase their creativity through contrast, so we built an effect that can help them accomplish that on Reels. We took inspiration from the anime art style popular in online culture today, and hoped that users could make use of our segmentation and appearance modification to build videos that resonate with their interests.
What it does
The filter starts with an on-screen portal, which disappears when the user interacts with it by moving into it. Then the world is replaced, the user is transported into a manga, and their facial features are replaced with the large eyes, mouth, and cheek features typical in anime media.
How we built it
Development Platform We built the project in SparkAR.
In the patch editor, this was accomplished by linking the script with a patch that measures less than or equal, and then creating visibility patches for each of the objects in the effect itself. When the value measured is less than zero, visibility is toggled "on".
The particle system utilizes a cherry blossom texture to make it appear as if cherry blossoms are emerging out of the portal.
The face objects were designed in Maya, utilizing reference images from the internet. These were then painted in Substance Painter, while the object itself was modified using the SparkAR toolkit in Blender. These objects were then assigned a material in SparkAR, and textured using the Substance Painter export.
The effect employs a face tracker patch, which is linked to eyeball, nose, cheek, and chin patches. The respective objects are linked by position and rotation to the tracker patches.
Challenges we ran into
We initially wanted to employ plane tracking to place the portal wherever the user desires, but found that it made the effect confusing to use. Perhaps future iterations will allow the user a more intuitive version of this idea.
Another serious challenge was the difficulty of being unable to correct object origins in SparkAR. We had to employ the Blender SparkAR toolkit to correct this, and then export from Blender. The user really should be able to modify object origins in SparkAR.
Accomplishments that we're proud of
What we learned
The most important thing we learned was how to utilize SparkAR to develop AR filter effects for Instagram. This development platform is powerful and offers a highly functional environment capable of producing complicate AR effects.
What's next for Anime Portal
The next direction for this effect is to improve and expand its functionality. The filter should be able to represent a range of emotions, eye colors, and react to the user closing and opening their mouth and eyes. We would also like to implement custom placement of the portal, perhaps even utilizing multiple portals to provide a range of effects. While we did not have enough time to implement all these features in this iteration of the filter, we hope to be able to expand its functionality in the future.