tryon.AI

Inspiration

The spark for Try On AI came from two completely distinct, real-world perspectives within our team: the frustrated online consumer and the retail frontline insider.

As customers, Espy, Ryan, Jonas, and Misa were tired of the guesswork inherent to online clothes shopping. Buying apparel from static 2D images often felt like a gamble, driving a strong personal desire for an intelligent system that could recommend coordinating items and remove the mystery of how clothes look before making a purchase.

On the flip side, Jonas brought crucial operational insights from his time working on the sales floor at Old Navy. They observed firsthand how physical fitting rooms bottleneck the brick-and-mortar customer experience and envisioned an interactive tool that could assist employees, streamline store operations, and elevate the hybrid shopping journey.

By merging the customer's need for real-time style curation with the retail worker's mission to optimize the shopper experience, we set out to build a platform that tackles an industry-wide challenge: the massive 20% e-commerce apparel return rate that drains merchant margins and frustrates buyers.

What It Does

Try On AI is an interactive digital smart mirror dashboard designed to bridge the gap between physical retail confidence and online shopping convenience.

When a user initializes the mirror, the interface hooks into their webcam using a high-fidelity video stream. From there, shoppers can browse a digital catalog of clothing options. Once an outfit is selected, the application captures the user's live video frame context and passes it alongside the target garment to an advanced generative image model. The image generation software then processes the attributes and instantly outputs a hyper-realistic visualization of the user wearing the exact product, rendering it seamlessly directly inside the mirror interface.

Going a step further, the integrated backend AI agent acts as an automated personal stylist, evaluating the user's style choices to instantly recommend matching complements—such as pants, jackets, or footwear—pulled directly from live online retail merchant catalogs.

How We Built It

We implemented a clean, decoupled monorepo architecture divided into two highly specialized layers:

Frontend Ecosystem (try-on-react): Built on React and Vite, utilizing customized glassmorphic CSS to craft a premium, futuristic HUD overlay complete with neon laser scanning effects. For the video infrastructure, we integrated the GetStream Video SDK to handle ultra-low latency real-time web streams seamlessly and capture stable user reference frames.
Intelligence & Generation Layer (my-agent): A separate backend agent directory designed to securely generate WebRTC authentication tokens, manage catalog syncing, and interface with the image generation API pipeline that handles the virtual clothing try-on transformations.

To accurately map, align, and position the generative canvas elements uniformly across varying viewport sizes, the application normalizes pixel data boundaries relative to the live camera canvas dimensions using standard layout constraints:

$$x_{\text{norm}} = \frac{x_{\text{pixel}}}{W_{\text{frame}}}$$

$$y_{\text{norm}} = \frac{y_{\text{pixel}}}{H_{\text{frame}}}$$

Where $W_{\text{frame}}$ and $H_{\text{frame}}$ represent the absolute width and height of the captured webcam media track, mapping any localized coordinates into a universal scaling format between $0$ and $1$.

Challenges We Faced

Synchronizing Video Feeds with Generative Pipelines: Transitioning from static overlays to on-demand image generation required a highly precise handshake. We had to optimize how we extracted high-quality reference frames from the live GetStream video feed and packaged them alongside apparel metadata to prevent generation timeouts and API latency from lagging out the user interface.
Maintaining Interactive Immersion: Because image generation software takes a moment to process and render a brand-new image, we had to design an interactive, premium "scanning" HUD animation sequence. This bridges the processing gap, keeping the user engaged and maintaining the psychological illusion of a futuristic smart mirror rather than a slow loading screen.

Accomplishments That We're Proud Of

Flawless Monorepo Separation: Successfully building and connecting an independent frontend repository with a detached backend agent server, establishing a foundation that mimics enterprise production architecture.
Premium Sci-Fi Aesthetic: Designing a highly functional, beautiful dashboard interface from scratch that incorporates frosted glass blurs, ambient glow states, and interactive HUD tracking elements without sacrificing page performance.
Seamless Hardware Lifecycle Management: Overcoming complex browser hardware bugs to transition into a robust, enterprise-grade video pipeline using GetStream to feed our image pipeline.

What We Learned

This project provided a deep dive into modern stream orchestration and generative AI pipelines. We realized that streaming infrastructure and image transformation models are incredibly difficult to link reliably from the ground up; leveraging dedicated tools like GetStream allows engineers to save massive amounts of time, freeing them up to focus on core features like the image generation workflow and recommendation engine rather than debugging raw WebRTC connection packets.

Additionally, we learned how designing technology simultaneously through the eyes of both a consumer and a retail worker results in a much more balanced, commercially viable product.

What's Next for Try On AI

The next step for our application is moving from on-demand single image generations to real-time generative video tracking. We plan to feed our GetStream video pipeline directly into a browser-based computer vision model like