FaceFit

A Personalized Face Mask constructed via Volumetric Regression Networks

Comment

Inspiration

The CDC, WHO, and many other prominent organizations in public health have said that the wearing of masks is crucial to preventing the spread of COVID-19. Currently, approximately 900,000 people worldwide face craniofacial deformities, making it far harder for them to wear a mask. Even if they end up wearing a mask, it is almost always a bad fit, defeating its purpose. To respond to this, we create FaceFit, a web application to build custom, personalized mask frames.

What is a mask frame?

A mask frame is an add-on to a traditional mask, which makes the mask a more snug fit. We included a picture of a traditional mask frame in the image gallery.

What it does

In the application, a user can scan their face. Using cutting-edge technology, including a VRM and AR, we construct a mask from a photo of one's face. Here is a basic control flow:

  1. User takes a picture of their face
  2. Image turned into 3D model via a VRM
  3. 3D model of mask frame created based on 3D model of face (Development still in progress)
  4. OBJ file of the model shared with the user
  5. User can see how the frame fits them with a mask using AR

How We built it

We used a Volumetric Regression Network (a VRM) to get a 3D model of one's face when given a 2D image, and connected this to our web app through the Flask framework. We also used echoAR to overlay a mask on oneself to give the user a feeling of how they would look when wearing a certain mask. We used the following tech stack:

  • Flask
  • VRM CNN Regression
  • HTML, CSS, and JS
  • echoAR

Challenges I ran into

Through time zones, it was difficult to communicate with the team, especially on an online platform. However, we pushed through, and we were able to make it. Also, many members of our learned echoAR for the first time, so we faced difficulties in learning and using echoAR.

Accomplishments that I'm proud of

We're proud of having a semi-function prototype that generates a 3D model given a picture of one's face, and having a web application deployed. We also are proud of finishing our pitch deck and cementing our idea.

What I learned

We learned how to use echoAR and how to integrate echoAR with Flask. In addition, we learned about computer vision and VRMs. A few of our teammates didn't have a technical background and others didn't have a biology background. We were able to pool our knowledge to support each other and we learned a lot from each other, given our diverse set of backgrounds.

What's next for FaceFit

We hope to complete the algorithm to generate personalized mask frames, and deploy that on a complete website. We plan on making money via selling this to mask manufacturers (3M) and individual hospitals.

Share this project:

Updates