Inspiration
A grocery listing can tell you a lot about a product, if you know where to look. A title might be abbreviated, the ingredients might reveal the answer, and the description might spend half its time advertising something else. We wanted to build a catalog agent that could work through that mess and improve when it got something wrong.
What it does
Our agent turns grocery listings into structured attributes for bread, cereal, soda, and yogurt. Take “Blood Orange Ginger Ale.” Blood orange describes the flavor, while ginger ale tells you what kind of drink it is. The agent uses the listing’s title, ingredients, category, and description to choose values from the challenge’s catalog.
How we built it
We built the extraction pipeline in Python. Each field stays separate so the agent can tell where a clue came from. We wrote specialized readers for cases where the wording matters, such as distinguishing different grains from different forms of the same grain.
Then we gave the agent labeled practice examples and a way to learn from its mistakes. It found recurring errors, proposed reusable rules, and tested those rules on separate product families. It saved rules that passed those checks to a checkpoint and could reuse them after a restart. We measured the effect of these saved rules separately from the improvements we wrote by hand.
How we used Databricks
We used a Databricks notebook and MLflow to review the agent’s learning experiments across 763 grouped practice products. We brought in the evaluation results and tracked each round’s micro-F1, correct values, extra predictions, missed values, and how many proposed rules were accepted or rejected. The notebook also saved a structured summary of each comparison.
That breakdown let us ask better questions than “Did the score go up?” We could see whether a round recovered missing attributes, removed unsupported guesses, or traded one kind of error for another. Tracking rejected rules also gave us a record of the ideas the agent tried and why they stayed out of the saved checkpoint.
Challenges we ran into
Sometimes the deciding ingredient is missing from the listing entirely. Other times the text gives you plenty of clues, but they describe a serving suggestion or a different product. We built mock listings to isolate those failures and checked whether the resulting fixes also helped on real practice products.
We also found that a rule can help on unfamiliar product families and have little effect on unfamiliar brands. Testing both splits gave us a clearer picture of where the agent generalized and where it still struggled.
Accomplishments that we're proud of
Our frozen agent generated valid predictions for every product in the organizer’s holdout. We used the organizer’s unchanged scorer, disabled learning during final inference, and reproduced the same predictions in a separate run.
Our favorite learning result was a bread classification rule the agent discovered from practice mistakes. It saved the rule and applied it again after a restart. In a controlled comparison on held-out practice families, the rule corrected more predictions than it broke. The experiment records show both the gain and where it failed to carry over.
What we learned
The biggest improvements came from understanding what a clue actually describes. Once the agent could distinguish a product’s identity from its flavor, or its own ingredients from a suggested topping, it could make better use of the text it already had. The learning experiments also taught us when to leave a proposed rule out.
What's next for DataJohn
We’d like to show the source of each prediction and flag cases where evidence is missing or conflicting. That would give catalog teams a short list of listings to review and show them exactly what needs a closer look.
Built With
- databricks
- gitlab
- python
