Inspiration
H-E-B's challenge: turn messy grocery product descriptions into structured attributes, and build an agent that teaches itself to improve. We framed it as learning a catalog's labeling conventions from examples, with a fixed base model, so all learning had to live around the model: its examples, its rules, and the corrections applied to its answers.
What it does
The agent labels every product attribute with Qwen3-Next 80B (served on Databricks), shown the most similar labeled products as examples. In its learning step it improves itself three ways, without changing any model weights:
- Rules: a teacher model (Claude Sonnet 4.5) reads the agent's own mistakes on held-back products and rewrites one guideline per attribute. A rewrite is kept only if it raises that attribute's F1.
- Learned corrections: the agent predicts every training product leave-self-out, mines phrase rules from the labels, and keeps only the rules that fix its own out-of-sample mistakes on both halves of the data.
- When to trust the neighbors: with attribute-specific similarity, it learns a threshold above which a strong majority of similar labeled products overrides the model.
How we built it
- Databricks Free Edition model serving (AI Gateway) for the allowed base model, across our workspaces, plus the organizers' Python harness for the agent contract, metering and validation.
- A local 5-fold cross-validation scorer using the official micro-F1, with a disk cache of model replies, so post-processing changes replay identical model answers and measured gains are real, not sampling noise.
- An AI-run research sweep (error anatomy, prompt audit, retrieval analysis, literature scan) produced ideas; every idea was tested on cross-validation and most were dropped.
Results
- Learning evidence (5-fold CV on the experiment data): Q 0.809 before learning to 0.900 after, with the gain in every fold.
- Hold-out (run once, organizer scorer unchanged): Q 0.8987; two declared reproducibility reruns gave 0.8971 and 0.8981.
Challenges
- Judging changed mid-event from a cost-weighted composite to accuracy plus learning evidence, so we pivoted from a cheap local-model hybrid to an accuracy-first design.
- Free Edition rate limits: we paced calls and pooled workspaces, and added retries so malformed model replies never drop a product.
- Small validation sets accepted rules that did not generalize; learning on all training products leave-self-out with a both-halves gate fixed most of that.
What we learned
- The model often knows the rule and still does not follow it. The biggest late gains came from learning corrections around the model, not from better prompts.
- Examples beat instructions: accuracy rises sharply with the number of shown examples that carry the right label.
- Measure everything on held-out folds; many plausible ideas (voting, correction memory, per-attribute calls) did nothing.
- Label noise sets a ceiling around 0.90 to 0.93, and we are close to it.
What's next
Verified rules (teacher claims checked against the data), larger internal validation, and a cleaner way to make the base model follow learned rules directly.
Built With
- databricks
- python
