sigma_tech

Conversational shopping agent that finds the right product in 10 questions, works offline, made to work in real world situations.

Comment

Inspiration

Problem Statement 4 challenged us to build a conversational shopping agent that truly understands customers, not just matching keywords, but asking the right questions at the right time to find their perfect product within 10 turns. We were inspired by the gap between how people naturally describe what they want ("something for Jiu Jitsu") and traditional search systems that demand exact product names.

What it does

Our agent conducts intelligent product discovery conversations:

  • Asks strategic clarifying questions (material, features, color, style, size, use case, budget) in an optimized order
  • Accumulates customer preferences turn-by-turn into an ever-refining search query
  • Never repeats failed suggestions, intelligently paging through fresh candidates
  • Detects contradictions when customers change their mind mid-conversation
  • Finds the target product in top-10 with 99.5% success rate in ~3 turns average

Works completely offline

How we built it

Core Architecture:

  • Intent Decoder (Tier 1): Regex-based frame matching against 8 known utterance patterns—not ML estimation, ensuring deterministic behavior
  • Constraint Ledger: Verbatim accumulation of every customer reply into the search query (drove 0.16→0.75 hit rate improvement alone)
  • BM25 Retrieval: SQLite FTS5 over 50K products with weighted field scoring
  • Seven-Slot Ask Policy: Fixed schedule for turns 1-7, adaptive fallback for 8-10
  • Never-Repeat Rule: Partition-based (not filter) to keep top-10 always full while excluding shown products

Development Stack:

  • Python 3.10+
  • 390 unit tests ensuring offline safety and contract compliance
  • Custom CLI server/client for manual conversation testing
  • Cached-trajectory replay system for rapid experimentation (turns 11-point sweeps from days into seconds)

Optional Layers (Disabled in Submission):

  • Cross-encoder reranking tested (+0.047 TechnicalScore but 1.2s/turn cost)
  • Semantic fallback (Tier 2) with embedding nearest-centroid for paraphrase handling
  • Gemini 3.7 Flash (Tier 3): LLM escalation for vague output requiring semantics
  • All seams designed to degrade gracefully, never crashing the pipeline

Challenges we ran into

  1. The Simulator Leak: 94.5% of simulator constraints were verbatim substrings of target product listings—BM25 was being handed the answer key. We discovered this made local scores misleading:
    • Phrase matching: +0.0588 locally, +0.0017 on real ESCI queries
    • Dense fusion: -0.11 locally, +0.06 on ESCI queries (suppressed improvement)

Solution: Dual-bracket scoring (leaky/scrubbed) + ESCI dataset (600 human-authored queries) as ground truth.

  1. The Never-Raise Rule: The evaluator silently converts exceptions to score=0. A single throw in init kills all 200 sessions, not just one turn.

Solution: Three-layer guarded construction (index/reranker/semantic), each with independent try/except. Degraded mode still returns valid responses with real ask_attributes.

  1. Dependency Minefield: Cross-encoder reranking showed +0.047 gain but required sentence-transformers, PyTorch, bundled weights—first external dependencies on graded path.

Solution: Designed as optional seam that checks flag BEFORE checking imports. Ships as NullReranker (identity function). Standard library only in production.

  1. Schedule vs. Classifier Mismatch: Fixed schedule asks budget at turn 7, but evaluator's classifier tests budget first. Only ~40/200 sessions reach turn 7 - structural misalignment

Solution: Turn 8-10 either asking an attribute that has not hit a useful information tap or asking the other attributes (Brand, Category, Other)

Accomplishments that we're proud of

Measurable Results:

  • 0.872 TechnicalScore (leaky) / 0.72783 (scrubbed) on 200 sessions
  • 99.5% hit@10 / 66% hit@10 (scrubbed—more realistic)
  • 19ms per turn (9.7s for 200 sessions, 571 turns total)
  • $0.00 API cost - truly zero, not "effectively zero" on offline pipeline

Rigorous Measurement Discipline:

  • Caught 3 measurement artifacts (phrase matching, dense fusion, cross-encoder headline mismatch)
  • Bootstrap confidence intervals on all deltas
  • Never quoted single-bracket scores without the pair
  • ESCI validation prevented overfitting to leaky simulator

Production-Grade Reliability:

  • Verified offline under sandbox-exec with networking revoked (scores identical)
  • 390 tests, 100% passing (CI explicitly names modules—new tests don't silently skip)
  • Degraded mode: missing catalog → still asks questions, returns schema-valid responses
  • Never crashes: respond() returns empty_response() on any exception

Architectural Clarity:

  • 18 modules in src/, clean separation: slots never touch retrieval (parsing bug can't corrupt search)
  • Seams for 4 optional layers, all inert by default
  • Single 200-line CLAUDE.md explains entire system to any teammate

What we learned

  1. The Simplest Idea Matters Most: Verbatim constraint accumulation (0.16→0.75) outperformed every ML enhancement by an order of magnitude. Sometimes the "boring" solution is the right one.

    1. Your Evaluation Rig IS Your Product: We shipped the never-repeat rule before measuring target rank distribution in missed sessions (debt documented in docs/todo.md). The structural proof of zero-cost let us ship early, but we still owe the effectiveness measurement.
    2. Measurement Artifacts Are Silent Killers: Local score alone would have shipped phrase matching (+0.0588) and rejected dense fusion (-0.11)—both wrong calls. ESCI queries reversed both verdicts. Never trust one instrument.
    3. Failure Modes Design Themselves In: The evaluator's silent-zero on exceptions made every design decision a security review:
    4. Module-scope imports must never raise
    5. init failures kill 200 sessions, not one turn
    6. Optional layers need independent guards (shared try-except lets broken reranker kill working index)
    7. Documentation Beats Memory: docs/todo.md captured 9 open decisions with evidence/seams/settlement criteria. Prevented decay and let anyone resume work cold.

What's next for sigma_tech

  1. Ask-Yield Adaptive Ordering:
    • Only ~40/200 sessions reach turn 7 (premise challenged)
    • Measure on that subset specifically, not diluted across all 200
    • Innovation play (20pts), not TechnicalScore delta
  2. Dense Fusion Revival:
    • ESCI shows +0.06 recall@10, suppressed locally by simulator leak
    • Dependency budget question remains (first external dep on graded path)
    • R4 weighted sweep never got bootstrap CI—compute that first
  3. Schedule Reordering:
    • Test frequency-ordered (budget first) vs hand-tuned (budget seventh)
    • Budget-classified constraints common, but we ask at turn 7 when many sessions ended
    • Human decision required, not delta alone
    • Dynamic scheduling of attributes based on assigned weights
  4. Real Intent Override Handling:
    • Current: shown-set guard (products before override go back in play)
    • Next: Ledger segmentation—pre/post override treated as separate queries
  5. Contradiction Recovery:
    • Detect slot-value contradictions (shipped)
    • Resolve them—ask "You said leather earlier, now cloth. Which matters more?"
  6. Explainability Layer:
    • "I'm recommending this because you mentioned waterproof + leather + hiking"
    • Build from ledger + BM25 match terms (data already present)
  7. Multi-Catalog Federation:
    • Current: 50K products, single catalog
    • Scale: Multiple verticals (electronics, fashion, home), cross-catalog deduplication

Built With

+ 10 more
Share this project:

Updates