SubStrata

SubStrata classifies GEE satellite tiles and scores predictions into a memory graph, so past mistakes shift future classifications without retraining, enabling lightweight yet accurate automation

Comment

Inspiration

We've worked at NASA, and the biggest issue that stuck with us was that even today, with land-cover and land-use datasets like Dynamic World and WorldCover feeding climate, agriculture, and disaster-response work, a huge amount of the actual classification and segmentation is still done, or manually corrected, by scientists sitting down and labeling imagery by hand. Not because the problem is unsolved in theory, but because the models good enough to do it reliably are too heavy, too slow, and too expensive to run at the scale and cadence agencies actually need: continuously, over huge swaths of the planet, on a budget that isn't a supercomputing grant.

The standard response to a model getting something wrong is fine-tuning: collect corrections, retrain, redeploy — exactly the kind of heavy, GPU-and-weeks cycle that keeps humans in the loop as the permanent fallback, batch after batch, mistake after mistake, with the model never actually getting cheaper or faster to keep correcting. We wanted to see if a much lighter system could close that gap not by getting bigger, but by getting smarter about its own mistakes.

What it does

https://youtu.be/j5m8wr0mu2c

SubStrata is a self-improving land-cover classifier that learns entirely through memory, not retraining. It runs a closed loop: classify a batch → score against ground truth → identify recurring error patterns → extract a plain-English heuristic → file it on a graph → retrieve the relevant heuristics before the next batch → repeat.

Every mistake gets analyzed, turned into a human-readable lesson, and stored in a graph structured by an is_a class hierarchy (trees, shrub, grass, and crops all roll up to vegetation). Because lessons are filed on hierarchy nodes instead of just the exact classes that produced them, they transfer — a heuristic learned from trees-vs-shrub confusion can fix a shrub-vs-grass confusion the system has never personally made, before it ever sees an example of it. The live dashboard shows error rates dropping batch over batch, side by side across three memory backends (no memory, raw retrieval, and our reflective graph), so the improvement is provable, not asserted.

What we built today, specifically for HackSprint:

  • Fireworks AI — we researched and tried multiple models and ended up using Fireworks-hosted Qwen 2.5 vision model as the classification backend. Qwen 2.5 is also paired with a specific LLM tool calling system that lets Qwen call a smaller LLM model specifically fine-tuned for image classification. Accuracy held up against our prior baseline.
  • Braintrust — we wired in a real evaluation layer that scores every batch's predictions through Braintrust instead of relying only on our own local accuracy math, and used its scored output to drive our ablation comparison. Across our ablation arms, the reflective graph scored 89% versus 64% for raw nearest-neighbor retrieval — a real, Braintrust-verified gap, not a number we computed ourselves.
  • CopilotKit — we built our live dashboard's heuristic-transfer panel on CopilotKit. Heuristic text now streams in live as the Strategist agent writes it, and a live graph shows the model's accuracy climbing as the memory layer accumulates lessons — both rendered as generative UI instead of a static post-hoc chart.
  • Daytona — we moved each batch's classify/score/update-graph sequence into an isolated Daytona sandbox, so a bad batch can't take down the host process.
  • CodeRabbit — connected to the repo and reviewing our commits from today.

The demo: proving it actually learns

This is the part we think is the most important to get right. Self-improvement claims are worthless unless they're shown happening, live:

  1. Batch 1 — a specific, recurring confusion pair (e.g. trees vs. shrub) appears with no heuristics yet in place.
  2. The loop runs — a heuristic is extracted in plain English and filed to the graph. We show it on screen, streaming in live via CopilotKit, in plain words, not a confidence score.
  3. Batch 2 — a held-out batch with the same confusion pair. The heuristic is retrieved, and the error rate on that pair visibly drops.
  4. The transfer beat — a later batch contains a never-seen confusion pair from the same family (e.g. shrub vs. grass). The system made no errors on this pair to learn from, yet the lesson transfers because it was filed at the right level of the class hierarchy.
  5. The ablation — the same patch stream run through three memory backends side by side: cold (no memory), raw retrieval (k-NN over past examples), and our reflective graph. Braintrust-scored results: our reflective graph hit 89% versus 64% for raw retrieval, and the graph is the only arm that fixes the never-seen pair — because retrieval alone has nothing to retrieve for a pair it's never seen.

Challenges we ran into

Getting a provable self-improvement, not just a plausible one, shaped almost every design decision in the original architecture — one ground-truth label per patch instead of a fuzzy mask, plain-English heuristics instead of embeddings alone, three ablation arms instead of one.

Today's biggest challenges were integration challenges: getting all of our sponsor APIs — Braintrust, CopilotKit, Fireworks, and Daytona — talking to each other without breaking. The trickiest part was the frontend-to-backend connection: our live SSE event stream had to be piped into CopilotKit's state layer without dropping or duplicating events, and we hit a few real bugs getting the heuristic-streaming panel to update in sync with the backend's batch loop instead of lagging behind or double-rendering. Working through those bugs live, under time pressure, was the real test of the day.

Accomplishments that we're proud of

We're proud that SubStrata doesn't force the usual tradeoff between accuracy and efficiency — no weights change, no retraining job runs, just a graph of plain-English lessons consulted before each classification.

We're especially proud of the result: a heuristic learned from one confusion pair measurably improved accuracy on a different, never-before-seen confusion pair, purely because the lesson was filed at the right level of the class hierarchy — and that result is backed by a real Braintrust score (89% vs. 64%), not just our own claim.

Beyond the architecture, we're proud of two things from today specifically: building something we genuinely believe is applicable to real companies and agencies dealing with this exact bottleneck, and ending the day with a tool that actually, functionally works end-to-end — not a demo held together by luck, but a pipeline we watched run, live, integrated across five sponsor tools in one day.

What we learned

That memory can substitute for retraining in a much more literal way than we expected — not as a cache of past answers, but as a structured, inheritable body of reasoning that a fixed-weight model can consult before it acts. We also relearned a more basic lesson: a system that claims to "self-improve" has to be built demo-first, around the exact before/after moment that proves it, or you end up with an architecture that's interesting but unprovable on stage.

What's next for SubStrata

The loop — classify, evaluate, extract, persist, retrieve — is domain-agnostic. The same architecture applies anywhere expert correction is the bottleneck on AI adoption and failure patterns repeat: boundary-precise area accounting for carbon markets (deforestation hectares), flood-extent estimation for insurance, medical imaging triage, or manufacturing defect detection. Memory-driven adaptation offers a path to systems that keep improving in production — auditable, and without the cost or latency of retraining.

Built With

Share this project:

Updates