TT Farm

Inspiration

A customer hides one product among 50,000 and gives the agent ten turns. Predictable customers can be handled quickly, but real conversations drift, override earlier preferences, and use unfamiliar language. TT Farm therefore combines two complementary decision systems and lets them share one case file.

What it does

TT Farm is an autonomous shopping copilot with a fast deterministic reflex layer and a deeper LLM agent. Routine turns are handled locally in milliseconds. A confidence controller continuously watches parser fallbacks, category-match quality, clue-harvest progress, and the run-level miss rate. When a conversation is losing in an unfamiliar regime and a live model is available, it hands the complete conversation state to the LLM agent. The two systems then alternate turns over a shared exclusion list until the product is found.

How we built it

A Python, standard-library-only reflex layer with typed conversational slots
Category-pool retrieval and evidence ranking over the frozen 50,000-product catalog
An autonomous two-stage LLM loop: plan local FTS5 searches, then rank only retrieved product IDs
A confidence controller for per-session escalation and run-level regime detection
Shared exclusions, intent-override decay, token/session budgets, and a governor that disables an unhelpful takeover
Strict JSON validation, candidate allow-listing, deduplication, and a fully offline fallback
The unmodified official evaluator for every reported public benchmark

This implements runtime workflow re-orchestration: the agent changes its retrieval and reasoning path dynamically instead of committing to one fixed pipeline.

Challenges we ran into

The deterministic and LLM paths win different sessions, so a full takeover can discard the fast layer's correct answers. Paired per-session analysis led us to alternate the two systems after escalation rather than replace one with the other. We also had to preserve prior evidence across intent overrides, negotiate differences between model APIs, and guarantee valid behavior when final judging disables network access.

Accomplishments that we're proud of

On all 200 released public sessions, the offline reflex path achieved TechnicalScore 0.9748, HitRate@10 1.000, MRR 0.995, and MTTC 2.185 with zero model tokens.

On a hostile stress harness with more human-like wording, the fused system achieved TechnicalScore 0.8100 and Hit Rate 0.905. It outperformed both the reflex layer alone and an always-on flagship model, while using less than one third of the latter's tokens. The LLM path alone was also evaluated on all 200 released sessions, achieving TechnicalScore 0.814001.

What we learned

Architecture matters more than simply choosing a larger model. Fast local structure is ideal for familiar cases, while model reasoning is most valuable when triggered by evidence that the current workflow is failing. Conversation state, candidate discipline, and measured handoff policies determine whether the combination actually improves performance.

What's next for TT Farm

We plan to improve semantic candidate recall, calibrate escalation thresholds on broader hidden-style conversations, reduce LLM latency and token cost, and expand the controller into a general policy for dynamically composing retrieval and reasoning tools.

Reproducibility

The public repository contains both agent paths, the confidence controller, setup instructions, evaluation disclosure, robustness tools, and aggregated results. The same Agent remains valid with or without API credentials.

Development tools used

Python 3.10+ (standard library only at runtime)
Git and GitHub using a fork-and-pull-request workflow, plus GitHub CLI
OpenAI Codex and Anthropic Claude Code as pair-programming agents for coding, experiment harnesses, and documentation drafting; every reported measurement comes from evaluation scripts committed in the public repository
Google NotebookLM for generating the demo video from the team's written project documentation
The official TechJam participant kit's unmodified local evaluator as the test harness
Headless Chrome for rendering the explainer PDFs

Team contributions

Ming Wang — Autonomous AI agent in starter/: LLM planning and ranking loop, model client, live evaluation runs, and repository ownership
Wenjie Huang — Reflex layer and escalation in ttfarm/: fast path, confidence controller, handover logic, evaluation and robustness tooling, and submission writing
Yang Xu — Early-stage brainstorming and track research, QA design, data preparation, and independent reproduction of the evaluation pipeline from a clean clone
jayde zhang — Early-stage brainstorming and pre-research, submission QA, demo-video fact-checking, documentation proofreading, registration and publication verification, and data-preparation support