K-Tryon

Inspiration

If you have went online shopping before, the most common frustration you would find is that you simply cannot try out the product you are about to buy. How would you tell whether or not it is suitable for your needs? After looking through research on AR, I stumbled upon VITON - Virtual Try-on. By using VITON, users will be able to find out how different clothes can fit on them, while looking at how it matches with the rest of their outfit.

What it does

K-Tryon is an entire e-commerce app, implementing the functionality of an enterprise online shop such as user authentication and payment systems. When you enter the app, you will see a list of recommended products based on the user's purchase history. After selecting an item of your liking, you will be presented with more details on the item, and an option to try on the item. To try on the item, simply send over a full body image of yourself (at least down to the knees). You can check back in after a minute or so to view the modified image!

How we built it

The product recommendation is trained using real enterprise data provided by H&M. This dataset gives us a list of items, with its related tags such as the type of clothing/unique features of the clothing, as well as a list of users and their purchase history.

A pretrained VITON model was used: VITON-HD. It presents itself as a "High-Resolution Virtual Try-On via Misalignment-Aware Normalization", a research project showcased in CVPR 2021. Unlike other market competitors, this VITON model utilises extensive preprocessing and settles for a higher computational cost, to render the output image with high precision and resolution.

Finally, a frontend for this application was built using android studio, while a flask server was made with sqlite to host the backend of the application.

Challenges we ran into

Although VITON-HD requires extensive preprocessing, the documentation for preprocessing is rather little. This resulted in us having to reverse engineer many parts of preprocessing. For example, we found out that the background colour of the clothing image should be #f6f6f6. For the CIHP_PGN program, I had to downscale the input image as I would run out of memory with a 768x1024 image. However, I was unable to find out how to correctly upscale the output image, which led to many inconsistencies in the output of VITON-HD.

Accomplishments that we're proud of

Despite some small issues in setting things up, we were ultimately able to make a working full-blown app. We are glad to have tinkered with these research projects and brought use to them by implementing them in an e-commerce app.

What we learned

In the training of the recommendation AI, we learnt to apply what we had learnt from the workshop to train an AI on our own. In the setting up of VITON, we learnt to have a perseverant mindset, especially when we are treading in fields of expertise we never had any experience in.