Self-Improving Catalog Agent on Databricks

Inspiration

Retailers like H-E-B fill in product attributes (cereal grain, yogurt type, soda features) for thousands of products from messy descriptions. Calling a large LLM for every product is expensive and ignores the retailer's own labeling conventions. We wanted an agent that learns those conventions itself, improves automatically, and proves every improvement. Databricks let us make that learning repeatable, measurable and governed.

What it does

The agent learns in six automated steps:

Memorize: builds an evidence memory of which phrases point to which values.
Critique and revise: tests itself on held-out products and rewrites its own decision rules.
Retrieve: learns when to trust similar past products.
Skills from mistakes: turns recurring errors into small correction rules.
LLM teacher: foundation models suggest cues for its mistakes.
Know when to ask: for products where its memory is unsure, it asks an LLM, showing it the most similar solved products as examples.

Every change is kept only if it improves accuracy on products the agent has never seen. Of 2,895 LLM-teacher suggestions, only 81 passed.

The score is exact-match micro-F1:

$$Q = \frac{2\,TP}{2\,TP + FP + FN}$$

On the organizers' hidden hold-out of 415 products, $Q$ rose from 0.852 to 0.878 (+0.026, 95% CI [+0.015, +0.038]). Our held-out experiments predicted +0.029 before we scored.

The agent then labels the whole catalog and sends only its least-confident predictions to a human review app. Corrections flow back as verified answers.

How we built it

Unity Catalog + Delta Lake: raw files land in a Volume and become clean tables. Every learning step, per-attribute score, learned skill and LLM suggestion (with its verdict and rationale) is a governed Delta row. Delta versioning keeps the agent's knowledge auditable.
Mosaic AI Gateway:
- five foundation models (qwen3-next-80b, gpt-oss-120b, llama-4-maverick, llama-3.3-70b, gemma-3-12b) act as interchangeable teachers, so we measured which teaches best
- qwen3-next-80b is also the base model the agent consults for uncertain products
Lakeflow Jobs (serverless): a 3-task pipeline: learn & evaluate → register & promote → label the catalog.
MLflow + Unity Catalog model registry: every learning run is tracked step by step. A champion/challenger gate lets a new agent version replace the old one only if it is better on held-out data. It has already refused a challenger that merely tied.
Databricks Apps: a Streamlit review app shows reviewers only uncertain predictions. Decisions land in a Delta table and are served as human_verified answers.
AI/BI dashboard + Genie: learning evidence at a glance. Genie answers questions like "why were most teacher suggestions rejected?" in plain English.
Git folders: the repo lives in the workspace, and the challenge harness reads training data straight from Unity Catalog.

Challenges we ran into

Rate limits: our first live run had 335 of 358 LLM calls rejected. We added retry-after backoff, time budgets and a circuit breaker; the next run had 2 failures.
Staying honest with a hidden test set: we froze every version and committed its predictions before scoring, and fixed our decision rules before measuring.
Knowing what not to learn: unrestricted skills learned template artifacts like "12oz". Restricting skills to real attribute evidence removed them at no cost in accuracy.
Overfitting our own improvements: a finer threshold grid looked promising but did worse on unseen splits, so our validation gate rejected it.

Accomplishments that we're proud of

A predicted gain that held up: +0.029 predicted from untouched splits, +0.026 measured on the hold-out.
Teachers measured, not assumed: a 12B model teaches almost as well as a 120B one. AI Gateway made that a measurement, not a guess.
A full loop on one platform: learn → validate → promote → label → human review → learn again, all governed in Unity Catalog.

What we learned

The memory and the LLM make different mistakes. Deciding per product when to trust each one is the key to accuracy.
Validation gates matter more than clever ideas: most proposals should be rejected.
Rate limits and failure handling are design constraints, not afterthoughts.

What's next

The agent writes its own per-attribute labeling guidelines (learned prompts), validated like every other lesson.
Contrastive examples for the values it confuses most.
A nightly Job on new products, with reviewers seeing only the uncertain few percent.