Inspiration
From our RnD, we realized that the fashion industry suffers from a number of challenges: some being an inconsistent supply of fabrics, poor online shopping experience for consumers where what you see is not what get's delivered, with no virtual fitting options. There is also the issue of products from high-end fashion brands being reproduced as counterfeits and sold online with the consumer having no means to know what real and what's fake. Case in point, nike seized selling its products on amazon. We solve some of these problems by looking at 3D commerce for fashion, and use social media to boost the virality of products.
What it does
KlurdyXR is an ai-powered experience for rendering 3D graphics. We are piloting with the fashion industry by providing virtual fitting experiences on social media. The models track human poses from a camera feed in real-time and use this inference to know where to augment the apparel on top of the person.
How I built it
We trained 3 different models separately to perform specific tasks, that is single person detection in a frame, body part segmentation, and single human pose estimation in the wild. These models need to be exported in onnx format. With the help of the snap team, we integrated these models using SnapML features on lens studio forming a machine learning pipeline using the device camera texture as the input. We modeled apparel sketches provided to us by a fashion designer into 3D assets, using a skinned body.with an occlusion body. Fabrics were digitized using photoshop, using 1024x1024 textures. After importing these assets in Lens studio, we created a script for swapping materials and post-processing output from ML pipeline to placing 3D assets in required targets on the screen. We opted to use world space features of lens studio to infer the depth of a coordinate in the frame.
Challenges I ran into
- SnapML has a limit of 10Mb model sizes, so we had to compromise the complexity of networks to have mobile-first neural networks *8 2D pose estimation is good but still misses the z-axis. We are working on 3D pose inference on mobile to make it better with videos
Accomplishments that I'm proud of
Our team is the first on the planet to ever produce and launch apparel virtual fitting experiences on snap chat. We are pioneers in running mobile XR fitting experiences in the fashion industry. .
What I learned
- PyTorch support for onnx makes models available on web-based platforms
- Optimization and fine-tuning of pre-trained PyTorch models
- Cloud plays an important role in accelerating ML experiments
What's next for Klurdy XR
- Integrate models with our web app and create an experience on the mobile web.
- Integrate models with unreal engine 4 and unity and commercialize as AR plugin
Log in or sign up for Devpost to join the conversation.