Inspiration
What drew us in was how much the scoring rewards precision. Each session gives the agent at most ten turns to find one hidden target among 50,000 products, and the provided starter agent scored only 0.10671. The session ends as soon as the target enters the top 10, and its rank is locked permanently on that first appearance. So we built the agent to narrow the candidate pool before revealing its strongest matches.
What it does
Fifty thousand products. One hidden target. Ten turns.
The agent talks to a shopper, accumulates what they say, and narrows the catalog until one product remains. It does this with no model, no network and no API key. The intelligence is in the index it builds before the conversation starts.
The offline build — once, at startup
We replay the evaluator's own card-building function over every product in the catalog, indexing what each product would say about itself rather than its raw text. That produces six lookup structures:
| Structure | What it does for robustness |
|---|---|
| Clue index | Whole preferences are matched exactly. A damaged phrase fails to match and falls through to the next step, so it never matches the wrong product. |
| Category buckets | The product type is locked from the first message. If it doesn't match a known type exactly, it's recorded as a guess, so later steps are allowed to look outside it. |
| BM25 word index | A keyword safety net that still works when preferences arrive reworded or with words missing. |
| Popularity + profile priors | A sensible ordering for turns where the conversation has given nothing to filter on yet. |
| Canonicalizer | Translates casual words into catalog words (violet to purple), consulted only when nothing else matched. |
| Dense encoder (optional, off) | Meaning-based matching for heavy rewording. Disabled by default behind the TECHJAM_DENSE environment variable, so scoring never depends on extra software or a network. |
Standard library only. Everything in memory. No network, no keys, no tokens.
Every turn — seven steps
1 · Understand. Reads the shopper's message, works out what they want, and notices when they change their mind. Anything not recognised with certainty is recorded as a guess rather than a fact, so one wrong reading can never lock the agent in.
2 · Remember. Keeps every preference heard so far. Nothing is discarded casually. A preference is dropped only when the catalog proves no product can satisfy both it and the newer one, and a mind-change downgrades the old product type back to a guess.
3 · Shortlist. Filters to the right product type, then to items matching every stated preference. The strictest filter runs first and falls back one level at a time. There is always a next step, so a misleading message can't leave the agent stuck or empty-handed.
4 · Order. Best matches first, with popularity and personal taste breaking ties. The same input always produces the same list, because a fixed tie-break makes behaviour reproducible and testable.
5 · Choose how many to show. Confident, and it shows the top pick. Unsure, and it holds the strongest candidates back so they can lead the next turn. Rank locks the instant the target appears, so a page of maybes costs more than it gains.
6 · Ask one question. One short question per turn keeps new information flowing. The asking stops once the shopper has nothing left, so no turn is wasted on a dead end.
7 · Answer. Up to ten products plus the question, wrapped in a safety net. If any step fails, the agent returns its last good list instead of an error, because an error forfeits the turn entirely.
The theme across all of it: the expensive or looser step runs only after the simple, precise one has provably failed.
Against the four pillars
I — Intent routing and hybrid pipeline. The six live routes, strictest first:
- Exact — intersecting every constraint heard so far, inside the category bucket, leaves exactly one product.
- Exact_global — the same intersection ignoring the category, when the category was fuzzy-matched rather than parsed exactly.
- Bm25_trust — constraints exist but none of them resolve in the clue index. BM25's ordering over the untouched pool is trusted instead of being overridden by an intersection that never narrowed anything.
- Wide — evidence is weak and the category was a guess. BM25 re-runs over the whole catalog with a boost for in-bucket items, and is adopted only if the separation between the top two candidates improves.
- Bm25 / Weak — the normal path. BM25 over the category bucket, marked weak when the top score or the gap to second place falls below threshold.
- Floor — no usable text at all. Order the bucket by popularity blended with profile-tag affinity.
Two further routes exist in source but are off in the shipped config: a dense lane behind the TECHJAM_DENSE environment variable, and an FTS floor that measured worse across our conditions.
II — Multi-turn scenario evolution.
On sessions where the shopper abandons one product and switches to another, the agent stranded itself in the old category. Hit rate on that axis was 0.493. Two fixes were candidates. Measured alone, demoting the pre-switch category to a guess was worth +0.084. Proving a contradiction catalog-wide before deleting a remembered preference measured at −0.001 and was nearly cut. Together they were worth +0.181. The demotion opens an escape route but leaves poisoned constraints in place; the proof clears them but has nowhere to send the agent. Neither solo measurement described the mechanism.
III — Dynamic context programming.
Raw dialogue never reaches the retrieval layer. Each turn we distil the message into structured state, maintain a certainty flag on the parsed category, and re-orchestrate which routes are eligible accordingly.
For the long-term profile half, the shopper's preference tags blend with popularity to order candidates on the evidence-free route: the opening of a browsing or boundary session, where nothing has narrowed the pool yet. We measured it in isolation across all six conditions:
| Condition | Prior on | Prior off | Δ |
|---|---|---|---|
| Clean, as shipped | 0.98000 | 0.97970 | +0.00030 |
| Clues: 20% of words dropped | 0.97524 | 0.97484 | +0.00040 |
| Clues: 35% dropped | 0.97590 | 0.97530 | +0.00060 |
| Clues: 50% dropped | 0.97510 | 0.97450 | +0.00060 |
| Category words reordered | 0.97930 | 0.97900 | +0.00030 |
| Category reordered + 35% drop | 0.94620 | 0.94560 | +0.00060 |
Positive on all six, and the gain runs entirely through time-to-conversion. Hit rate and MRR are already saturated at 1.000, so MTTC is the only channel left open. It moves 2.015 to 2.000, and the effect sits where the mechanism lives: boundary sessions gain 0.20 turns, browsing 0.01, buying and intent_override nothing at all.
The honest caveat is that this rests on three changed sessions out of two hundred. The direction is clean, with six of six conditions positive, three of three changed sessions improved and no regressions. But with n=3 a different draw could show nothing. It is a real, reproducible, correctly-signed effect of small magnitude, not a load-bearing one.
That smallness is itself a product of the saturated benchmark. With hit rate and MRR both pinned at 1.000, the ceiling on any ranking improvement is whatever MTTC can still move. On the private 800, where those metrics are unlikely to be saturated, the prior may be worth more than +0.0006, though that is an inference and not something we can measure.
IV — Evaluation matrix.
Coverage, precision and efficiency. Full results in Accomplishments.
Why this matters beyond the benchmark
Conversational product discovery is normally built on per-query model inference, so every shopper conversation carries a cost, a latency budget, and a dependency on an external service staying up. This agent answers in a fraction of a millisecond, costs nothing per user, needs no GPU and no credentials, and returns identical results for identical input. It is cheap to run and fully auditable at the scale of a real commerce surface, where inference cost rather than model quality is usually the binding constraint.
The transferable part is the architecture, not the index. Route strictest first, escalate only on a proven failure signature, never discard what the shopper told you without proof, and never guess when you can ask. That pattern applies to any catalog where structured attributes sit alongside free text, which is most of retail. A production deployment would swap our exact-phrase index for the merchant's own attribute data and keep every other layer intact.
How we built it
Architecture
The build step runs once at startup. Rather than indexing raw product text, the agent replays the evaluator's intent_card() function over all 50,000 products and indexes the output, which means the index holds the exact constraint sentences a shopper can utter. Agent vocabulary and shopper vocabulary are equal by construction, and that is what makes exact matching viable. One pass builds the six structures in the table above. The clue index maps 60,670 distinct constraint strings to the products that emit them, and 91.3% of those strings identify exactly one product, so a single remembered preference is usually a near-decisive filter. The category buckets split the catalog into 1,115 product types. Everything lives in memory; nothing is persisted or downloaded.
Every turn runs seven stages, the technical counterparts of the seven steps above. Parse classifies the message by template shape, and anything it cannot recognise with certainty is recorded as a guess. Remember accumulates constraints verbatim, and a constraint is only discarded by the catalog-wide contradiction proof described under Pillar II. Narrow runs the retrieval cascade, strictest route first, with looser routes armed only by an explicit failure signature such as a repaired category or a constraint that resolves nowhere in the clue index. Rank sorts candidates by evidence strength, breaks ties by popularity and finally by parent_asin, and on evidence-free turns falls back to popularity blended with the preference-tag profile prior. Identical input always produces identical output. Schedule assigns candidates to (turn, rank) slots by the value the official metric gives each slot, which is how a strong runner-up ends up held for a rank-1 slot next turn instead of shown at rank 2 now. Fixed caps limit turn 1 and evidence-free turns to a single card. Ask emits one structured question per turn and stops once the shopper has nothing left to give. Reply wraps the whole turn in a never-raise guard that returns the last good page on any internal fault, because a raised exception scores the turn as empty.
Development tools used
Developed across team laptops (macOS and Windows), with no cloud compute or GPU. Reported performance figures were measured on a MacBook Pro (Apple M4, 16 GB RAM) running CPython 3.11.12, and independently cross-checked on a second machine under conda CPython 3.13.5.
- Editors / IDEs: Visual Studio Code
- AI-assisted development: AI coding assistants such as Claude Code and Codex were used throughout for implementation, code review and measurement tooling. All architectural decisions, experiments and measurements are our own, and every claim in this write-up is backed by a run of the organizer's evaluator.
- Version control: git, with a public GitHub repository
- Coordination: Telegram, with a written policy requiring every pipeline change to be announced to the team before merge
Alongside the agent itself we built our own measurement tooling, included in the repository:
| Tool | Purpose |
|---|---|
tools/verify_agent.py |
Runs the agent across all corruption conditions in one pass and emits the full robustness table |
tools/smoke_test.py |
Validates the agent against the organizer's API contract before any measurement is trusted |
tools/config_sweep.py |
Sweeps configuration constants to confirm we are on a plateau rather than a spike |
tools/bakeoff.py |
Runs two agent variants through the same sessions and conditions, so an adoption decision is a measurement rather than an argument |
tools/stress10k.py |
Generates 10,000 synthetic sessions across five difficulty types the agent was never designed for: natural rewording, vague descriptions, damaged category names, shoppers who switch to an entirely different product, and rarely-reviewed targets. It reports hit rate, ranking quality, turns-to-find and score for each |
tools/turn1_suite.py |
Builds 1,000 sessions where the opening message alone identifies the product uniquely, then reports how often the agent delivers it on turn one, a direct test of the best-case speed limit |
All scoring reported here comes from the organizer's own unmodified evaluator/local_evaluator.py. We verified that evaluator/, data/ and docs/ are zero-diff. tests/ contains our own additional suites alongside the organizer's. Our tooling surrounds that evaluator; it never replaces it.
APIs used
None.
The agent makes no network calls, requires no credentials, and consumes zero tokens, verified as 0 prompt and 0 completion tokens in the evaluator's own output. Estimated model cost is zero, because there is no model.
Network access and external API calls are permitted for final scoring, and an offline fallback is not mandatory. Running without either is therefore a deliberate engineering choice rather than a constraint we were handed. No credentials to manage, no service availability risk, no per-query cost, and nothing that can vary between runs.
We built and benchmarked model-based retrieval and ranking, and it lost:
| Approach | Score | vs. agent at time of test |
|---|---|---|
| Dense embedding retrieval (bge-small, bge-base, gte-base), fair test with query prefix | 0.922 | −0.055 |
| Cross-encoder reranking | 0.953 | −0.024 |
| Agent at time of test (no model) | 0.977 | — |
| Current shipped agent | 0.980000 | — |
Nine rankers were evaluated in total. The cross-encoder actively degraded results. The reason is structural: our retrieval stage collapses the candidate pool to a median of one product by turn two, so we measured the ceiling on perfect reranking of the residual at 0.006. A hosted model would have added cost, latency, non-determinism and an external dependency for six thousandths of a point.
A dense-retrieval path remains in the source but is disabled by default behind the TECHJAM_DENSE environment variable. It is measured and not shipped, not a feature.
Libraries and frameworks used
Python standard library only. No third-party dependencies, no package installation step.
Modules used: json, math, re, difflib, collections, sqlite3, pathlib.
Startup cost is about 6 seconds, incurred once. Median per-turn latency is 0.26 ms (p95 2.9 ms, worst 96.8 ms). Memory is about 724 MiB after the index build and about 938 MiB peak across a full 200-session run; that peak is process-wide and includes the evaluator's own copy of the catalog. Build plus all 200 sessions completes in 6.8 seconds wall clock. Measured on CPython 3.11.12, MacBook Pro (Apple M4, 16 GB RAM), no GPU or MPS used. On a slower laptop the same build takes about 12 s with a 0.6 ms median, so treat startup as machine-dependent.
Frameworks evaluated and not adopted: sentence-transformers and torch, covering three embedding models and two cross-encoders. Measurements above.
The practical consequence is that a reviewer can clone the repository and reproduce our reported score with no environment setup beyond a Python interpreter. We verified this: a fresh clone to a reproduced score of 0.98 took 13 seconds, following only the README.
The run is byte-identical across differing PYTHONHASHSEED values, verified under seeds 0, 1337 and 12345. We treat this as load-bearing rather than incidental. Before we enforced an explicit tie-break in every sort, run-to-run variance was ±0.03 and swamped every measurement we were trying to make.
Datasets and assets used
Organizer-provided, exclusively. No external corpora were introduced.
- Product catalog: the frozen 50,000-product set (18.3 MB compressed, 58 MB unpacked) from the Amazon Reviews 2023
Clothing_Shoes_and_Jewelrycategory, supplied in the participant kit and verified against the published SHA256 checksum. Used strictly read-only, with no mutations and no synthetic ASIN injection. - Evaluation sessions: the 200 labelled public development sessions. The 800 private sessions are held by the organizer and were never accessible to us.
- Evaluator: the organizer's
evaluator/local_evaluator.py, unmodified.
Derived assets, built at runtime from the catalog alone. At startup we replay the evaluator's own constraint-generation procedure across all 50,000 products and build the six lookup structures listed above: clue index, category buckets, BM25 word index, popularity and profile priors, canonicalizer, and the optional dense encoder, which is off by default.
Synthetic evaluation sets.
Beyond the 200 public sessions we generated 11,000 of our own, deterministically seeded and reproducible from the repo.
stress10k produces 10,000 sessions across five axes, 2,000 each: realistic paraphrase (24 rewrites plus 15% random word loss), vague description where "nylon" becomes "that slick sporty material", damaged category names, true product switches where the opening message describes a different product entirely, and cold targets drawn without any popularity filter. All targets exclude the 200 public ones.
turn1_suite produces 1,000 sessions in which the opening message is by construction enough to identify the product uniquely: 383 with a unique raw feature string, 383 with a rare material or colour in a small category, and all 234 categories in the catalog that contain exactly one product. The messages are entirely clean. The adversarial part is target selection only.
On independence. The paraphrase and vague-description vocabularies share no tokens with the agent's own 160-entry canonicalizer, so those transforms cannot be matched by a table we wrote. One axis, true product switch, was used to select between agent variants, and the constants it settled are cited in our config. The remaining four axes were used for measurement. turn1_suite was written after the final change to the agent, so it is post-hoc verification and could not have been tuned against.
Challenges we ran into
Our measurements were lying to us
For the first day of building, every A/B comparison we ran was noise.
The agent sorts candidate sets, and those sorts had ties. Python's sort is stable, so tied elements fell back to set-iteration order, which varies with PYTHONHASHSEED between processes. Run-to-run variance was ±0.03, larger than most of the effects we were trying to measure. We were confidently adopting and rejecting changes based on differences that did not exist.
The fix was one line: an explicit parent_asin tie-break in every sort. Finding it required noticing that a result looked too clean, rather than looking wrong. Determinism has been treated as load-bearing ever since.
A second instance of the same class of bug lived in SQL rather than Python. SQLite gives no ordering guarantee for rows with equal bm25() scores, so that path could vary across builds even with a fixed hash seed. A fixed PYTHONHASHSEED would never have caught it.
Three independent agents, merged on evidence
Three of us developed separate agents in parallel before combining the strongest parts into one submission. Rather than choosing ideas based on preference, we evaluated every agent using the same sessions and corruption conditions through tools/bakeoff.py. Each comparison ran twice, to check that an apparent improvement was reproducible.
That process led to two measured adoptions. Germaine's category-free clue intersection improved the score by +0.0487 under category damage and +0.0194 when category damage was combined with 35% clue dropping, while leaving the other four conditions unchanged. Marcus's one-card rule for evidence-free turns improved all six tested conditions on the current agent by about +0.001, including +0.00115 on clean sessions, and contributed to browsing and boundary MRR reaching 1.000.
A path bug that could fail the entire run
The agent originally resolved the catalog path relative to the current working directory. If the evaluator started it outside the repository root, initialization raised a FileNotFoundError. The evaluator's error handling covers respond() but not agent construction, so this would fail the entire run before the first turn and score zero.
We caught it by testing agent initialization from different working directories. The agent now searches a fixed list of module-relative catalog locations, while any path explicitly provided by the caller is used unchanged. Testing respond() alone was not enough: initialization and reset() are part of the scored execution path too.
There was nothing to film
The brief puts UI development out of scope, so the agent is backend-only and runs in a terminal. That left a submission requirement, a video demonstrating the solution working end to end, with nothing to point a camera at.
We built a read-only viewer that runs the real agent on a real session and prints its internal state turn by turn: constraints accumulating, the candidate pool collapsing, which retrieval route fired, and what was held back. It lives outside the submission package and cannot reach the scored path.
Building it surfaced two defects in how we described ourselves. The viewer initially labelled already-shown candidates as "held", conflating two different mechanisms, and a judge counting cards across turns could have contradicted the screen. It also revealed that a docstring in our scheduler described behaviour the code did not have.
Accomplishments that we're proud of
Results
Measured with the organizer's unmodified evaluator across all 200 public sessions.
| Metric | Result |
|---|---|
| Hit Rate@10 | 1.000 |
| MRR | 1.000000 |
| MTTC | 2.00 turns |
| TechnicalScore | 0.980000 |
| Starter baseline | 0.10671 |
200 of 200 sessions convert at rank 1. The worst rank across the entire set is 1. Every scenario reaches MRR 1.000, across browsing, boundary, intent_override and buying alike.
| Scenario | MRR | MTTC |
|---|---|---|
| buying | 1.000 | 1.525 |
| browsing | 1.000 | 1.812 |
| boundary | 1.000 | 2.300 |
| intent_override | 1.000 | 3.667 |
Turn distribution: 78 sessions convert on turn 1, 72 on turn 2, 27 on turn 3, 20 on turn 4, 2 on turn 5 and 1 on turn 7. Zero misses.
One number worth stating plainly, because it bounds how much of this the conversation actually earns. A popularity-only floor, using none of the conversation at all, already reaches 0.832 on the same 200 sessions. The conversation is worth the last 0.148.
Robustness
The public 200 sessions measure whether the agent works. They do not measure whether it will keep working on 800 sessions we cannot see, and a perfect score on a development set is by itself the strongest possible signal of having fitted to it. That is why the number above is not the evidence we lead with.
We corrupted the input and report every condition rather than a single figure.
| Condition | Score | HR@10 | MRR | MTTC |
|---|---|---|---|---|
| Clean, as shipped | 0.98000 | 1.000 | 1.000 | 2.00 |
| Clues: 20% of words dropped | 0.97524 | 1.000 | 0.993 | 2.13 |
| Clues: 35% dropped | 0.97590 | 1.000 | 1.000 | 2.21 |
| Clues: 50% dropped | 0.97510 | 1.000 | 0.997 | 2.19 |
| Category words reordered | 0.97930 | 1.000 | 1.000 | 2.04 |
| Category reordered + 35% drop | 0.94620 | 0.970 | 0.970 | 2.49 |
| Starter baseline | 0.10671 | 0.125 | 0.068 | 9.81 |
Half the clue words destroyed costs 0.005. Damaging the category and the clues together costs 0.034, and even then hit rate stays at 0.970.
Reproducibility
python3 -m evaluator.local_evaluator
Verified from a cold clone: 13 seconds from cold start to a reproduced score of 0.98 once the catalog is in place, following only the README. results.json is byte-identical across differing PYTHONHASHSEED values, verified under 0, 1337 and 12345.
What we learned
We tested the four pillars instead of assuming every component would help
We implemented and evaluated the brief's proposed components, then kept only those that improved performance. That led us to diverge in three areas: dense retrieval, model-based ranking and slot erasure during intent overrides.
Ungated dense retrieval reduced the clean score by 0.018. A gated version improved paraphrased inputs by 0.036, which did not justify enabling the extra retrieval lane by default, so it remains optional behind an environment flag. We also excluded model-based reranking, because every reranker we tested performed worse than the existing cascade: 0.922 for dense embedding retrieval and 0.953 for the cross-encoder, against 0.977 for the agent at the time of testing. Finally, instead of erasing all previous preferences after an override, we retain those that remain compatible and remove one only when the catalog proves it contradicts the new requirement.
The four pillars were useful areas to investigate. Each implementation still had to justify its place through evaluation.
Compatible preferences can survive an intent override
The brief recommends clearing previous preferences when the shopper changes direction. In the evaluator, though, the abandoned preference and the new demand are both generated from the same target record. The earlier preference is therefore still factually true of the target, and discarding it removes useful evidence.
Our agent keeps the earlier preference unless the catalog proves it contradicts the new requirement. A contradiction exists only when each preference matches products independently but no product in the full catalog satisfies both. The previous category is also downgraded from a fact to a guess, which lets retrieval search outside it. Intent-override sessions reached MRR 1.000 and an MTTC of 3.667 turns, which makes the scenario the brief identifies as hardest the one the current agent solves completely.
Ranking had almost nothing left to do
The retrieval cascade reduces the candidate pool to a median of one product by turn two, leaving little for a more complex ranker to improve. A perfect reranker applied to the remaining candidates had a measured maximum upside of 0.006.
This explains why dense retrieval, model-based ranking and personalization all provided limited value. They were competing to improve the same small residual after retrieval had already resolved most sessions. On a catalog with less distinctive constraints and larger candidate pools, those components would have more ambiguity to resolve and would likely matter more.
What's next
Limitations
- A perfect public score does not establish private-set performance. Our 0.980 score and MRR 1.000 were measured on the same 200 public sessions used during development. With all 200 targets now appearing at rank one, this set can no longer distinguish between candidate designs, even though configuration decisions were made over differences as small as 0.0003. Performance on the 800 private sessions remains unknown.
- Our exact-match retrieval is closely coupled to the evaluator. We reproduce its constraint-generation procedure and index the exact phrases the simulated shopper can emit. Real shoppers would not consistently describe products in catalog wording, so this component would contribute less in a live system. The strict-to-loose retrieval architecture transfers; its dependence on evaluator-generated phrases does not.
- The popularity prior depends on how target products are sampled. It helps when little evidence is available, but it may produce the wrong ordering if the private target distribution differs from review-count popularity.
- Our robustness tests vary clue text and category wording while keeping the same evaluator and session-generation process. The hardest tested condition scored 0.94620. The organizer's final evaluation FAQ states that the final sessions use the same input schema, metric formula, stopping rule and deterministic customer-message templates as the released evaluator, with no undisclosed natural-language paraphrases introduced. Our corruption tests are therefore a margin of safety beyond the stated conditions rather than a substitute for an untested harness. The genuine remaining unknown is the target distribution: 800 unseen products, which is what our cold-target testing was built to bound.
What we would do with more time
- Test independently written conversations. We would ask human testers to describe products without seeing their catalog wording, then measure how well the agent performs when exact phrase overlap disappears.
- Broaden the robustness suite. We would test spelling errors, irrelevant details, conflicting preferences and repeated intent changes, while keeping the test vocabulary separate from the agent's own lookup tables.
- Separate overlapping scheduler effects. The evidence-free one-card rule currently improves the six tested conditions by about 0.001, but its effect overlaps with the later turn-one cap. We would run a full factorial ablation to isolate the contribution of each mechanism.
- Evaluate long-term profiles properly. The current
preference_tagsprior improved the clean score from 0.97840 to 0.97870, with gains of roughly 0.0003 to 0.0006 across all six conditions. We would evaluate it across multiple sessions per shopper, where persistent preferences could provide stronger evidence. - Revisit dense retrieval on a harder catalog. The gated dense lane improved paraphrased inputs by 0.036, while ungated use reduced the clean score by 0.018. We would test it on catalogs with less distinctive constraints and larger candidate pools, where semantic retrieval has more ambiguity to resolve.
Team — Star Labs
Nothing shipped on argument. Every mechanism below beat the measurement on a shared test battery first, so each name here corresponds to code that survived that gate.
Jovan — Team lead. Designed the architecture and the evidence-gated retrieval cascade. Built the intent-override handling, which demotes the pre-switch category and proves contradictions catalog-wide and is worth +0.181 on true product switches. Also built the lexical canonicalizer and the 10,000-session and turn-1 stress suites. Owned integration and regression verification. Found the determinism bug, where untied sorts made every A/B comparison noise, and the one-line fix for it is what made the rest of our measurements trustworthy.
Germaine — Lead tester, and the source of several shipped mechanisms. Generated thousands of adversarial sessions built to break the agent. Contributed the slot-value scheduler, the category-free clue intersection, the turn-1 one-card cap that took clean MRR to 1.0000, and the rule that trusts BM25's ordering when no constraint resolves, which is our single largest robustness gain. Also caught the working-directory bug that would have scored a whole run as zero.
Ben — Assisted with architecture and pipeline design, and ran testing. Researched paraphrase recovery: recovering constraints from reworded messages, a denormalizer for conversational phrasings, and word-count category matching. It lifts our worst-case template condition substantially. Wrote the unit-test suite for the agent's turn policy.
Marcus — Assisted with pipeline flow design and contributed the evidence-free one-card rule, which improved all six corruption conditions and took browsing and boundary MRR to 1.000. Found that a failure in reset() is unguarded by the evaluator and would abort a whole run. Audited the write-up against the code before publication and caught stale performance figures and an overstated claim. Drove the Devpost submission.
Pinwei — Owned how the work reads to anyone outside the team. Contributed to pipeline ideation, then shaped the submission: the elevator pitch, the framing of the project description, and the sections explaining why the problem matters and who the agent is for. Storyboarded the demo video, wrote and narrated the script, and set the order the story gets told in: the problem, the two pipelines, a live session, the results, then what we got wrong. Our agent runs in a terminal with no interface, so every choice about what a viewer sees and when was his.
Built With
- bm25
- conversational-ai
- git
- github
- information-retrieval
- inverted-index
- natural-language-processing
- python
- python-standard-library
- recommender-systems
- search
- sqlite
- zero-dependency
Log in or sign up for Devpost to join the conversation.