StyleSwap Shopper — Try On, Then Watch AI Buy It Live

An AI stylist that lets you virtually try on clothes using Nova Canvas, then watches a ghost browser buy them for you in real-time with Nova Act — all powered by Bedrock AgentCore.

Comment

Inspiration

Online shopping is broken. You scroll through hundreds of products, guess how they'll look on you, order three sizes hoping one fits, and return the rest. We wanted to ask: what if AI could close that gap entirely — let you see yourself wearing any outfit before you buy, and then handle the purchase for you while you watch?

The release of Amazon Nova Canvas (for image generation), Nova Act (for browser automation), and Bedrock AgentCore gave us the perfect stack to build this end-to-end. We saw an opportunity to combine these into a single conversational agent that acts as a personal stylist — one that doesn't just recommend clothes, but shows you wearing them and shops on your behalf with full transparency.

What it does

StyleSwap Shopper is an AI-powered fashion assistant where you:

  1. Chat with a personal stylist agent about what you want to wear
  2. Upload a photo of yourself — the AI analyzes your body type using Nova Lite
  3. Share any product link — the agent scrapes product details and downloads images
  4. See yourself in the outfit — Nova Canvas generates a photorealistic virtual try-on image
  5. Watch the AI buy it for you — Nova Act launches a ghost browser that navigates Amazon, selects your size, adds to cart, and proceeds to checkout — all streamed live to your screen via AWS DCV
  6. Review what happened — after the session ends, step-by-step screenshots show exactly what the automation did

The entire experience happens in a single chat interface with a split-panel layout: conversation on the left, live try-on images and ghost browser on the right.

How we built it

The architecture is fully serverless on AWS:

  • Agent Runtime: Bedrock AgentCore with Strands SDK for the conversational agent, using Nova Pro for reasoning
  • Virtual Try-On: Nova Canvas image generation with the COLOR_GUIDED_GENERATION task type for realistic outfit overlay
  • Photo Analysis: Nova Lite for body type estimation from user photos
  • Purchase Automation: Nova Act with AgentCore Browser for AI-driven browser automation
  • Live View: AWS DCV Web Client SDK integrated directly into the React frontend for real-time browser streaming
  • Infrastructure: CDK-deployed stack with Cognito auth, DynamoDB for profiles/history, S3 for images, API Gateway + Lambda for APIs, and Amplify for frontend hosting
  • Development: Built entirely using Kiro IDE with AI-assisted development, spec-driven design, and steering rules for code quality

The purchase flow uses a two-tool pattern: start_purchase_session returns the live view URL immediately so the frontend can connect via DCV, then execute_purchase runs the Nova Act automation while the user watches in real-time. After the session ends, the ghost browser tab transitions to a screenshot gallery showing each automation step.

Challenges we ran into

Nova Act Page.goto Timeout: Amazon product pages are heavy and would timeout during NovaAct's 30-second Playwright initialization. We solved this by starting on amazon.com homepage (fast load) and navigating to the product URL inside the act() instruction instead.

Streaming the Live View URL Early: Strands tools are synchronous — the tool result only reaches the frontend after the entire function completes. We initially tried async tool streaming (yield), but NovaAct's Playwright has thread affinity issues with asyncio.to_thread(). The solution was splitting into two sequential tools so the live view URL arrives before automation starts.

Parsing Python Repr Strings: Strands sends tool results as Python repr strings (single quotes, Python objects) through the SSE stream, not valid JSON. Standard JSON.parse() fails silently. We built a custom indexOf-based string extractor that reliably pulls field values from the Python dict format.

DCV SDK Integration: The AWS DCV Web Client SDK isn't an npm package — it's a downloadable archive that must be hosted as static files. Neither the ESM nor UMD version exposes a window.dcv global. We discovered the UMD version does set the global when loaded as a regular script (not type="module"), and integrated the full authenticate → connect → render pipeline with SigV4 parameter extraction from presigned URLs.

Screenshots URL Streaming Race: The screenshots JSON URL streams token-by-token through SSE. Early partial matches (e.g., Expires=177) would trigger 403 errors from S3. We fixed this by requiring exactly 10 digits after Expires= (Unix timestamp format) before dispatching the fetch.

What we learned

  • AgentCore's browser tool with DCV streaming is genuinely impressive for building transparent AI automation — users can watch and trust what the agent is doing
  • Nova Canvas virtual try-on produces surprisingly good results with minimal prompt engineering
  • The Strands SDK's tool system is flexible but the SSE streaming format requires careful handling on the frontend
  • Splitting long-running tools into start/execute pairs is a practical pattern for streaming intermediate results to users
  • Building with Kiro's spec-driven approach kept the architecture clean even as complexity grew rapidly during the hackathon

Accomplishments that we're proud of

  • End-to-end AI shopping in a single chat: From "I like this shirt" to seeing yourself wearing it to watching a ghost browser buy it — all without leaving the conversation. No other demo we've seen connects virtual try-on and automated purchase in one agent flow.

  • Live ghost browser streaming with DCV: We integrated the AWS DCV Web Client SDK directly into our React app to stream the browser automation in real-time. Users don't just trust the agent — they watch it work. Getting DCV authentication with SigV4 presigned URLs working from scratch was a significant technical achievement.

  • Seamless transition from live view to screenshots: When the browser session is active, users see the live DCV stream. The moment it ends, the same panel automatically fetches the Nova Act JSON and displays step-by-step screenshots. No page reload, no manual action — it just flows.

  • Two-tool pattern for early URL delivery: We solved the fundamental problem of streaming intermediate results from a long-running tool by splitting purchase automation into start_purchase_session (returns instantly with live view URL) and execute_purchase (runs the automation). The user sees the ghost browser before the first page even loads.

  • Built with Kiro from start to finish: The entire project — from spec design to CDK infrastructure to React frontend to agent tools — was built using Kiro IDE with AI-assisted development. Spec-driven workflows and steering rules kept the codebase consistent even as we iterated rapidly through dozens of deployments in a single hackathon session.

  • Production-grade architecture: This isn't a notebook demo. It's a fully deployed application with Cognito authentication, CDK infrastructure-as-code, Bedrock Guardrails for safety, AgentCore Memory for conversation history, and Amplify-hosted frontend — all deployable with a single cdk deploy.

What's next for StyleSwap Shopper — Try On, Then Watch AI Buy It Live

  • Integrate voice interaction using Nova Sonic for a hands-free styling experience
  • Add multi-product comparison — try on several outfits side by side
  • Support more retailers beyond Amazon
  • Add size recommendation using the body analysis data
  • Enable session replay from browser recordings stored in S3
  • Add Ghost browser control takeover/release feature for handling human input

Built With

  • agentcore-browser-tool
  • agentcore-gateway-(jwt-auth)
  • agentcore-memory
  • amazon-api-gateway
  • amazon-bedrock
  • amazon-bedrock-agentcore
  • amazon-bedrock-agentcore-runtime
  • amazon-cognito
  • amazon-dcv-web-client-sdk
  • amazon-dynamodb
  • amazon-nova-lite
  • amazon-nova-pro
  • amazon-nova-sonic
  • amazon-novaact
  • aws-amplify
  • aws-cloudformation/cdk
  • aws-iam
  • aws-lambda
  • bedrock-agentcore
  • python
  • react
  • strands
  • typescript
Share this project:

Updates