Refinery

Inspiration: ML teams spend days manually hunting for the images that would fix a failing class, then blindly re-augment the whole dataset when that doesn't work. We wanted an agent that figures out why a data fails and fixes it with evidence, rather than educated guesses with Chatgpt or Claude.

What it does: Refinery's main task is to completely automate the process of finding gaps and improving datasets. It audits a dataset, trains a baseline model, and pinpoints exactly where it's failing. It tries several fixes at once, trusted data, image transforms, web scraping, AI generation, verifies each candidate before trusting it, retrains, and only keeps a fix if it's proven to help. A dashboard shows the full run history (baseline vs. final recall, per-class gains) and every accepted dataset version, with the improved dataset downloadable as a ZIP.

How we built it: A Python pipeline (audit → diagnose → fix → retrain → verify), PyTorch models, Claude for reasoning about which fix to try and why, and Zero for policy-gated web search and image generation. A React frontend with Runs and Datasets tabs visualizes results and lets you download the patched dataset. Tested end-to-end successfully on MNIST.

Challenges we ran into: The hardest part wasn't finding a fix, but it was proving the fix was real. Small-data training is noisy enough that a "fix" can look great on one random seed and worse on another. We had to build gates that catch that noise instead of getting fooled by it.

Accomplishments that we're proud of: A run that took overall macro recall from 66.7% to 86.7%, a +20 point net gain, fully verified, with the patched dataset (train replaced, validation/diagnostic/test kept original and locked) ready to download. On a separate run, the agent lifted a starved class from 67.5% to 92.5% recall, and when that same fix caused a small dip elsewhere, it refused to publish and flagged it for a human instead of shipping a silent regression.

What we learned: Proving a fix works reliably is harder than finding one that seems to work. The verification and safety gates ended up mattering more than the remediation techniques themselves.

What's next for Refinery: More features aimed at handling worse starting points, messier, more broken datasets and weaker baseline models, so the agent can prove itself on harder problems, not just easy ones. We also want to implement a feature where the AI agent is given autonomy to email or call to garner access to more datasets that may be behind paywalls or educational institutions.

Built With

• anthropic-api
• cifar-10
• claude
• efficientnet
• imagenet
• mnist
• numpy
• pillow
• pydantic
• pytest
• python
• pytorch
• react
• recharts
• resnet
• shadcn-ui
• tailwindcss
• tanstack-router
• torchvision
• typescript
• vite
• zero