ShapeUp

Inspiration

ShapeUp started with our founders getting a bad haircut. We all know the type of haircuts that requires one to wear a hat for 2 weeks. So, we decided to build something to allow users to fully customize their hairstyle in a hyper-realistic 3D environment before getting a bad haircut.

What it does

ShapeUp lets a user preview and edit a hairstyle on a personalized 3D model before getting the haircut itself.

The flow is:

ShapeUp scans your face.
We use the single-view scan of your face to do point cloud estimation on a 3D, bald version of your head, and also run a separate algorithm to construct just your hair.
We render your head and hair in three js.
The user enters an editing loop, where they are able to describe the haircut they want in text or speech, and we will run our models to update the 3D haircut render.
The final look is summarized into a barber-readable spec the user can screenshot and bring to the shop.

Instead of relying on a generic style picker, ShapeUp gives the user a haircut preview on their own head geometry and lets them refine it interactively.

How we built it

We built Jerry by implementing algorithms from various research papers, namely, Facelift for the 3D head point cloud estimation, Hairstep for the hair model construction, and PERM for augmenting a dataset of 343 hairstyles with strand blending methods. We also performed PCA on this 343 hairstyles dataset to allow the user to adjust principal components of their hairstyle.

What we learned

Latency is very important. Because of our complex pipeline, we spent a lot of our time optimizing the speed for the user experience, including caching files, dropping slow models, etc..
3D hair rendering is extremely complex. Even after getting the gaussian splat, we spent a lot of time making our hair rendering realistic, since we had to render thousands of strands of hair from individual points and vertices.
The simplest solution is often the best one. Initially, we had a much more complicated code logic, and ended up dropping most of our early ideas. For example, we realized that instead of running a machine learning model each time for hairstyle generation, we could use a hair blending algorithm (PERM) to blend pre-computed hair models.

What's next for ShapeUp

We see several natural next steps:

Replace the preset-driven hair system with a learned deformation model for smoother continuous editing between styles.
Improve the export flow into a true barber-ready spec with measurements and view snapshots.
Improve hairstyle classification with a stronger curated taxonomy and a dedicated constrained style classifier.