FriendAR

After rounds of refinement still got much work to do

Inspiration

There are multiple occasions that during friends meet-up, there will be one or two there will not be able to make it. A lot of the time, we print those faces of friends that don't make it and use it for a group picture. We have lots of fun taking those pictures. What if we could use AR to replace those printed A4 paper? This is especially true during this pandemic where gathering are not encouraged. Able to virtually spawn a friend on your screen is awesome! On top of that, our team is span across 3 different countries, 3 different timezones. We are having fun hacking AR to make it seems like we are working together in the same location.

What it does

When you turn on your Instagram filter, it supposes to upload a picture of a person and translate it into AR and appear virtually next to you. For hackathon purposes, it will spawn Dara and Margo to Tien so it seems like we are working together for the hackathon.

How we built it

We build it with sparkAR. We also used Blender for our 3d model.

Challenges we ran into

3d modeling is hard while using our mediocre laptop, it becomes very toasty and time-consuming to render it. Besides that, we are having an issue with moving two objects on the screen. Another huge challenge from letting us build the solution we envision is we have no idea how to enable user to upload any picture and convert that picture into a 3d model on the fly.

Accomplishments that I'm proud of

Able to complete this challenge despite have 0 knowledge in SparkAR. On top of that, we are a team of 3 different countries from 3 different time zone!

What we learned

SparkAR is an awesome tool creating AR filters for Facebook and Instagram.

What's next for FriendAR

We hope that we could at least get some acknowledgment from Facebook hackathon team. We will continue to improve our 3d model to make it looks more realistic. We would like to explore the solution we envision which is to enable the user to upload any face/whole body image and convert it into a 3d model on the fly for the filter. Ideally, we foresee the following project milestones: We need to introduce the possibility to select a single or several photos of a friend from the user collection. An automatic or semi-supervised algorithm such as Graphcut[1] or a modern segmentation net (such as mobilenet) will extract the person from the collection of the images. A 3D avatar model is to be created then, using some recent works which propose single-shot 3d avatar creation [2]. Then the obtained 3D model can be used in our AR project.

Friends are always near!

References

Vicente, Sara, Vladimir Kolmogorov, and Carsten Rother. "Graph cut based image segmentation with connectivity priors." 2008 IEEE conference on computer vision and pattern recognition. IEEE, 2008.
Li, Zhong, et al. "3D Human Avatar Digitization from a Single Image." The 17th International Conference on Virtual-Reality Continuum and its Applications in Industry. 2019.