Virtual Try-on Luxury (VTL)

Experience fashion, virtually.

Comment

Inspiration

Throughout the pandemic, virtual try on has become a topic of interest among everyone. Online shopping for clothes is very fascinating as we can find different products and brands from various sellers all of them in the same place. Choosing the appropriate designs, sizes, and fits may make or break a purchase. This brings the issue of trying on garments to light, as dressing rooms are the focal point of real-life shopping trips. With technology that can realistically clothe an individual virtually, can make re-creating the in-store experience online achievable. Our team was intrigued by this concept, so we created our own extension: ‘VTL', where you can virtually try on clothes and experience fashion at its best.

What it does

Using our chrome extension VTL, users can select their favorite clothes and virtually try them on. VTL is really about the experience, and we believe that a better experience—one that provides a more realistic visualization of clothing—will boost buyer confidence, allowing them to make an appropriate fashion decision.

How we built it

Our solution comprises of these major modules:

  • Pose Generation using openpose
  • Generate Mask of Clothes using OpenCV bitmasks
  • Generate parsed image of user using a “Single Human Parser” pretrained model with a squeezenet backend
  • Generate Mask of person using OpenCV bitmasks and Pretrained squeezenet
  • Used Generative Adversarial Networks to generate the image of user with wearing the clothing item
  • Utilized a GMM (Geometric Matching Module) model to generate the warped clothes and warped cloth masks according to the target human and TOM (Try-On Module) model to blend the warped clothes from the GMM into the target human, to generate the final try-on output
  • Created a Pipeline to sequentially process the user’s image and the cloth to try on and generate the final try-on image.
  • Created a HTTP server with Flask to accept POST requests and send back the final try-on image
  • Created a chrome extension for users to virtually try on any piece of clothing that they see online

Challenges we ran into

The recommended model to use to generate the parsed human images is the Part Grouping Network. However, we weren’t able to set up the repository and use the pretrained weights. Hence, we looked for alternative pre-trained models (eg: squeezenet) to generate the parsed human images

We didn’t have access to GPUs and due to lack of time we couldn’t train it for many epochs. Hence, we looked for pre-trained models which haven’t been trained on our domain dataset for the project and the results aren’t perfect but they are good.

We faced issues in combining multiple preprocessing steps in the pipeline. This was very time consuming but we were able to debug the errors and fix the code.

Accomplishments that we're proud of

We were able to develop an error free full stack application that solves a business problem.

We were able to understand a research topic and utilize open source code to replicate the results.

What we learned

Advanced Deep Learning Algorithms (PoseNet, HumanParser, Geometric Matching Module, Try-On Module) How to create a Chrome extension

What's next for VTL

Extend the application to support multiple clothing types (trousers, masks, accessories, etc)

Make a VR platform to give the users a more in-shop experience

Built With

+ 36 more
Share this project:

Updates