Shopping Copilot w/ Accessibility Retreival

Conversational shopping copilot with a focus on accessibility-aware retrieval and ranking.

Comment

Inspiration

E-commerce search often fails at the intersection of vague intent and rigid keyword filters. Customers frequently enter stores with open-ended needs—such as "something comfortable for summer" or "a gift for my mom"—without knowing the exact technical attributes to search for.

I was inspired to build a shopping assistant that operates like an intuitive in-store associate: capable of understanding messy user language, asking targeted clarifying questions, and actively reasoning over nuanced accessibility requirements without overwhelming the customer.

What it does

Shopping Copilot w/ Accessibility Retrieval is a conversational product discovery agent that turns multi-turn, ambiguous dialogue into precise recommendations.

  • Conversational Constraint Accumulation: Tracks and updates customer preferences across multiple dialogue turns instead of treating queries in isolation.
  • Proactive Clarification Engine: Dynamically asks follow-up questions only when disambiguation improves retrieval quality.
  • Accessibility-Aware Matching: Balances standard product specifications with implicit and explicit accessibility preferences.
  • Interactive Web Demo: Features a lightweight browser UI that visualizes real-time candidate retrieval, confidence scoring, and dialogue state management.

How I built it

The project is built on a modular Python architecture designed for fast, local execution without excessive overhead:

  • Intent & Extraction Pipeline: Classifies user intent (browsing vs. buying) and extracts slot constraints from unstructured text.
  • Retrieval & Fusion Layer: Implemented using numpy and scikit-learn to index product catalogs and perform multi-signal candidate retrieval.
  • Reranking Workflow: Combines dynamic constraint satisfaction, preference scoring, and accessibility filters to surface top-ranked items.
  • Frontend & Demo Interface: A lightweight HTTP server paired with a vanilla HTML/CSS/JavaScript interface for real-time interaction.

System Pipeline

  1. Parse user intent and identify missing attributes.
  2. Update persistent state across turns.
  3. Retrieve candidate items from the catalog.
  4. Rerank candidates against accumulated constraints.
  5. Trigger clarification prompts if ambiguity exceeds threshold; otherwise, return final recommendations.

Challenges I ran into

  • Precision vs. Conversational Friction: Balancing when to ask clarifying questions versus when to show recommendations within strict multi-turn limits.
  • State Coherence: Retaining context and updating superseded preferences over multi-turn conversations without hallucinating outdated constraints.
  • Catalog Latency: Designing a lightweight, deterministic retrieval pipeline capable of scoring thousands of items rapidly in local environments.
  • Optimizing for Metric Trade-offs: Aligning agent behavior with quantitative retrieval metrics:

$$\text{TechnicalScore} = 0.50 \times \text{HitRate@10} + 0.30 \times \text{MRR} + 0.20 \times \text{Efficiency}$$

Accomplishments that I'm proud of

  • Built a functional end-to-end conversational agent that balances structured retrieval metrics with natural conversation.
  • Developed a clarification strategy that minimizes mean turns to convergence (MTTC) while boosting HitRate@10 and MRR.
  • Successfully unified multi-turn state management, accessibility retrieval, and candidate reranking into a lightweight Python stack.

What I learned

  • In conversational search, the core challenge is not just item matching, but deciding what to ask next to reduce uncertainty fastest.
  • Product thinking drives technical value: simple, deterministic retrieval loops often outperform overly complex architectures when optimizing for both speed and user trust.
  • Incorporating accessibility preferences requires dynamic constraint weighting rather than hard binary filters.

What's next for Shopping Copilot w/ Accessibility Retrieval

  • Multimodal Search: Integrating image input so users can upload photos alongside text descriptions.
  • Voice-First Navigation: Expanding accessibility by adding screen-reader-optimized outputs and bidirectional voice interactions.
  • Vector Hybrid Search: Upgrading the indexing layer with hybrid dense-sparse embeddings for deeper semantic catalog discovery.
Share this project:

Updates

Submission history