Magic Mirror

• 

  Promotional HTN Project Piece

Inspiration

In the digital landscape of the 21st century, we realized a core issue in online shopping: the gap between expectation and reality. The inability to physically try on clothes creates uncertainty in sizing, fit, and overall appearance, creating frustration and anxiety. After doing a market analysis of potential competing products, we realized that there is no publicly accessible solution that can feasibly be used. This challenge inspired us to re-imagine the online shopping experience, aiming not just to empower shoppers with better decisions but to enhance their confidence in the choices they make. Magic Mirror bridges the gap between the digital and physical worlds, allowing consumers to visualize how they’d look in an outfit before making the purchase. It's not just about finding the perfect fit; it’s about making sure every shopper can see themselves in the style they aspire to—making shopping a more informed and delightful experience.

What it does

Magic Mirror offers two transformative features that redefine the online shopping journey:

In-Depth Product Insights: Shoppers can access a comprehensive overview of product reviews, ratings, and summaries. By utilizing natural language processing, Magic Mirror curates detailed insights from a variety of sources. This empowers shoppers to not only make informed decisions but also feel confident in their choices. The feature fosters transparency, enabling businesses to build long-term trust and customer loyalty by offering genuine, data-backed insights.

Virtual Try-On: This core functionality allows users to upload an image of themselves and see how the clothes they are considering purchasing would look on them. Using advanced AI techniques, including image segmentation and inpainting with stable diffusion, Magic Mirror overlays garments with high fidelity, giving users a realistic preview of their future look. This not only reduces shopper regret but also contributes to fewer product returns, benefiting both consumers and businesses. A satisfied customer is more likely to return, making this feature essential for driving long-term business growth.

Behind the scenes, Magic Mirror runs a powerful computational pipeline, sorting and analyzing vast amounts of product information. The AI-driven system also outlines the pros and cons of each item. By seamlessly blending product insights and cutting-edge virtual try-ons, Magic Mirror turns a once-daunting decision into a delightful, confidence-building process.

How we built it

Our journey began by designing a sleek, intuitive user-interface using React, ensuring an engaging user experience that is both accessible and visually appealing. On the back-end, we deployed FastAPI, which served as the backbone for handling dynamic content, user uploads, and external API requests.

For the product insights, we leveraged Cohere’s LLMs (Large Language Models) to extract and summarize relevant product information, such as reviews and ratings, from across the web. This feature arms shoppers with all the information they need, compiled neatly in one place.

The virtual try-on functionality was built using open-source AI models. By integrating Stable Diffusion for image generation, we created a way for users to visualize their outfits with high levels of realism. This process involves AI-powered segmentation and in-painting to ensure the virtual try-on experience feels authentic and trustworthy.

Lastly, we developed a demo Shopify storefront and used Shopify’s API to dynamically pull products into Magic Mirror for demonstration purposes. The goal was to showcase how this technology could seamlessly integrate with any e-commerce platform. Throughout this process, we iterated over both the front- and back-end components, continuously refining them to ensure a smooth and polished user experience.

Challenges we ran into

Dev environments: Managing the dependency-heavy Python projects for the machine learning models was a significant challenge. Setting up and maintaining virtual environments while managing package dependencies became a top priority to ensure seamless development.

Compute times: Running complex image generation models initially led to long compute times, often in the range of hours. To optimize this, we optimization techniques to compress runtimes down to minutes.

Complex integration: Integrating the multi-component front-end and back-end posed significant hurdles. Communication between these two layers required rigorous testing and debugging, particularly as we introduced third-party APIs. Ensuring the various services worked in tandem while maintaining a consistent user experience required continuous iteration.

Device agnosticity: Although modern tools like React streamline the front-end development process, ensuring the interface maintained a consistent look across different devices was complex.

Accomplishments that we're proud of

Realistic Image Generation: Successfully implementing a machine learning pipeline that allows users to generate realistic, custom images of themselves wearing potential outfits. The level of realism is key in making Magic Mirror a valuable tool for consumers.

Aesthetically Pleasing Front-End: We’re particularly proud of our thematic, branded front-end that blends fashion sensibility with user-friendliness. The polished interface ensures that users engage confidently with the product.

Seamless API Integration: Bringing together diverse systems—AI image generation, product reviews, and Shopify’s API—into one cohesive, fully integrated platform is a significant technical feat.

What we learned

The Importance of Virtual Environments: Managing Python environments efficiently is crucial when working on machine learning projects with multiple dependencies. Proper environment isolation simplified the development process and ensured fewer compatibility issues.

Value of Design-Driven Planning: Focusing on both aesthetic and functional design from the start helped to create a product that is not only visually appealing but also functionally robust. A well-designed interface amplifies the impact of the technology behind it.

Power of a Singular Focus: Having a clear, singular vision allowed us to remain on track. Focusing on creating a virtual try-on tool while integrating key features like product insights helped us create a more cohesive and user-driven experience.

What's next for Magic Mirror

Enhanced Image Generation: We aim to further refine our image generation model, improving the overall realism and supporting more complex clothing items, such as layered outfits, jewelry, and accessories.

Chrome Extension for Seamless Integration: One of our milestones is the development of a Chrome extension for Magic Mirror. This would allow users to interact with the platform directly while browsing their favorite e-commerce stores. This streamlined integration will make the try-on process more convenient and accessible, expanding Magic Mirror’s usability.

Personalized Accounts with Trained LoRA Models: We are working toward creating an account system that allows users to save preferences, store images, and refine the accuracy of try-on results. In future iterations, users will be able to generate their own LoRA (Low-Rank Adaptation) models to create more personalized and realistic virtual try-ons.

Shopify Integration and Deployment: The next phase involves optimizing the app for Shopify, aiming for seamless integration that can be published on the Shopify marketplace. This will open the door for thousands of online stores to offer virtual try-on services, enhancing the experience.

Built With

• cohere
• fast-api
• huggingface
• machine-learning
• python
• react
• shopify
• stable-diffusion