ReStyle Buddy

Quick Idea Overview

This submission is about "presets" or "templates" or "plans" for style, grooming and makeup, how they can be created and later used to check for adherence. This works with either light or heavy makeup.

Inspiration

Over 900 million people cope with visual impairments, ranging from moderate to severe blindness. Of which over 510 million people have near vision problems. This includes people of all ages and gender and especially people above the age of 50.

Facing the challenges, millions of visually impaired people work and study in all fields. Day to day activities such as getting ready for work or school by light grooming or light makeup means following a set pattern of well practiced actions that lead a particular appearance and style. This range of appearances and styles that can be accomplished independently is limited. Also once or twice during the work-day, independently one would like to check that nothing is amiss or out of place.

The need to check that the appearance and styling that one is wearing has gone according to plan is important. One would also ideally like to consult an expert and follow advice with minimal followup.

What it does

The solution to this need of checking independently and learning with minimal followup is the ReStyle Buddy. This app an AI powered mobile application that can provide independence and empowerment in learning and expanding the skills needed to wear a new appearance and style. Upon starting the app, we are presented with two major functionalities - one regarding style presets, the other regarding grooming and makeup action presets. The style preset functionality allows the visually impaired person to capture a style which is an image with some assistance. At a later point in time the same person can use the style preset to verify that the appearances and style they are wearing actually matches a given preset. Similarly, the grooming and makeup actions preset can be used for instance by an Aveda expert consultant to record the right way of using a beauty product while instructing the same to the visually impaired person. This recording of actions is stored in the app and can be later used by the visually impaired person to learn and verify that the actions she is performing matches the instructions from the expert.

How we built it

To capture a style preset we perform a series of ML inference actions on the image that is streamed. We use face detection, followed by face and hair segmentation, the resultant masked out image is stored as a preset. In the Style check part in addition to face detection and segmentation we use a MobileNet v3 based model to generate an image embedding. This embedding of the current look of the subject is compared with the embeddings of all the previous presets. The top two matches are called out to the subject as the closest matches. Similarly in the grooming preset capture we use streaming pose detection to capture the 3D coordinates of the relevant upper body landmarks for a fixed number of image frames and save it. In the grooming check mode we compare the 3D coordinates of the current actions with the stored ones by using the Dynamic Time Warping algorithm. Based on a threshold we decide whether the current grooming actions compares well with the preset.

Please Note:

1. For a visually impaired person the volume-down button takes them to the "Style-Check" functionality and the volume-up button takes them to the "Grooming-Check" functionality from the home screen
2. Once the check is done the volume-up button takes them back to the home screen of the app.

Challenges we ran into

Data for training specific ML models! or rather lack of sufficient relevant data. We had to decide to use models trained on non-makeup data, with minimum impact on the accuracy of the predictions.

Accomplishments that we're proud of

The overall design of the app where we use the right ML capability to deliver value to the end-user. All the ML functionality in the ReStyle Buddy app is on-device. No external API's are used to deliver the ML functionality. Also the latency of the ML inference using CPU only is also impressive. Integrating with an Android native version of the DTW algorithm which is extremely performant has also been done.

What we learned

It is a matter of spending dedicated time and energy to solve problems using current tech so that it helps everyone in society. Where there is a will to help, things will happen for the good.

What's next for ReStyle Buddy

Data, data, data for building more accurate ML models !! If given the opportunity of accumulating and using specific image data (facial makeup images), then the accuracy of the Style Preset check functionality can be improved. Similarly given the opportunity of using specific human upper body motion data relevant to makeup application. The accuracy of the Grooming preset check can be improved by creating an accurate ML classifier.

Built With

• android
• camerax
• kotlin
• mlkit
• mobilenet
• tflite