Inspiration
This one's genuinely true, not written for effect: building this exact project tonight was the inspiration. Every time the agent hit a wall in this build — GitHub's /issues endpoint quietly hiding closed issues, the grading logic once matching a stray number buried in a sentence instead of the actual answer, a task blowing through its tool-call budget mid-reasoning — the fix always came the same way: look at what actually went wrong, write down the specific lesson, don't make that mistake again. That's exactly what every developer does when they're learning a new API for the first time. Track 1 asked for an agent that "genuinely gets better at a task over time" through its own self-reflection — the honest answer is that's just the debugging loop every one of us already lives in, and this project is that loop, formalized and made visible, running on the agent instead of the developer.
What It Does
The agent is handed one job: answer factual questions about a real GitHub repository by calling GitHub's live REST API. The first attempt gets several wrong, for real, verifiable reasons — GitHub's API defaults to open-only results unless you explicitly ask for everything, and the agent doesn't know that yet. A cheap classifier model diagnoses each failure into a specific reason, writes a one-sentence lesson to a growing memory.md file, and the same agent tries the same questions again — this time with its own notes in front of it. This repeats across multiple rounds. A live dashboard shows the whole thing happening in real time: the current round, the current question, the memory file growing line by line, and the success rate moving between rounds — not a replay, a genuinely live run.
How We Built It
- AO orchestrated every part of the build, across roughly 9 separate sessions — scaffolding, the GitHub tool wrapper, the evaluation suite, the round-running loop, the failure classifier, the memory system, the FastAPI backend, the frontend dashboard, and the final clean verification run.
- A custom GitHub API wrapper (
github_tool.py) that authenticates and correctly follows pagination via theLinkheader. - A frozen, hand-verified 10-question evaluation suite with ground truth computed once against a seeded repo, never touched again after that.
- GLM-4.7-Flash, served free via TensorMux, as the model actually attempting each round's tasks with real tool-calling.
- GPT-5 Nano as a cheap, fast classifier that diagnoses why each failure happened, into a fixed set of categories.
- Neatlogs tracing every round so the full tool-call history is inspectable, not just the final answer.
- A local FastAPI server exposing the run's live state as JSON, polled once a second by a plain HTML/CSS/vanilla-JS dashboard — no framework, no build step, nothing that could silently go stale.
Architecture
┌────────────────────┐
│ GitHub REST API │ (real, live, unmodified)
└─────────▲──────────┘
│ authenticated calls, auto-paginated
┌─────────┴──────────┐
│ github_tool.py │
└─────────▲──────────┘
│ exposed as a callable tool
┌───────────────────────────┴───────────────────────────┐
│ runner.py (the round loop) │
│ loads prompts/system_vN.md → gives task → agent calls │
│ the tool → grades the FINAL ANSWER line → logs result │
└──────────┬─────────────────────────────────┬────────────┘
│ model calls │ writes every step
┌─────────▼─────────┐ ┌────────▼─────────┐
│ GLM-4.7-Flash │ │ state.json │◄── polled every 1s
│ (via TensorMux) │ └────────▲─────────┘
└─────────────────────┘ │
│ on failure │ served as JSON
┌─────────▼─────────┐ ┌────────┴─────────┐
│ classifier.py │ │ server.py │
│ (GPT-5 Nano) │ │ (FastAPI) │
└─────────┬───────────┘ └────────▲─────────┘
│ appends a lesson │ HTTP
┌─────────▼─────────┐ ┌────────┴─────────┐
│ memory.py │ │ web/ dashboard │
│ → memory.md │ │ (HTML/CSS/JS) │
│ → system_v{N+1}.md │ └─────────────────────┘
└─────────────────────┘
│ fed back into the NEXT round's system prompt
└──────────────────────────────────────────────► (loop repeats)
Every arrow in this diagram is a real, live, running connection during the demo — not illustrative shorthand for something pre-recorded.
Challenges We Ran Into
Being fully honest, because these were the actual, real bugs, not smoothed-over ones:
- A grading bug that would have invalidated every result if we hadn't caught it. Our first pass at comparing the agent's answer to the expected value used a regex that searched for the target number anywhere in the free-text response — which meant a task where the agent said "20" but the text also happened to mention "40" elsewhere (describing unrelated seed data) was marked as a pass. We fixed this by requiring the agent to end every response with an unambiguous
FINAL ANSWER: <value>line and grading only that. - A silent crash-on-failure bug. Early on, any single task hitting its tool-call budget, or any HTTP error from the GitHub API, killed the entire round instead of just failing that one task. We rebuilt the loop so a single task's failure is recorded and the round continues.
- A genuinely authentic API gotcha, twice. GitHub's
/issuesendpoint and its/pullsendpoint both default to showing only open items unlessstate=allis explicitly passed — the agent hit this blind spot on both endpoints independently, in real, unstaged runs, which became two of the actual lessons written intomemory.md. - An encoding artifact where a trailing stray character after a correct number caused a technically-correct answer to fail extraction.
- Running on modest hardware — a mid-build crash from running two AO sessions simultaneously taught us to run sessions one at a time from then on.
Accomplishments We're Proud Of
- A pipeline that's genuinely re-runnable from a clean state and produces honestly different (not hardcoded, not replayed) numbers every time.
- Catching a subtle grading bug before it silently invalidated the whole project's core claim.
- A live dashboard that shows the actual mechanism — memory growing, rounds improving — happening in real time, not a mocked-up UI.
- A design that deliberately avoids the generic "AI-generated" look (no default gradient hero, no templated SaaS card grid) in favor of something plain and grounded in what the tool actually is: a running log of an agent's own trial and error.
What We Learned
- The most dangerous bugs in an agent-evaluation system aren't in the agent — they're in how you decide whether it was right, and that layer deserves as much scrutiny as the agent itself.
- Real third-party APIs have real, well-documented defaults that are easy for both a human and an agent to miss on a first attempt — and that's genuinely useful material for a self-improving system, not just a source of demo failures.
- Building this ourselves, with AO, mirrored the exact loop the project is about: try, fail for a specific reason, write it down, do better.
What's Next
- Generalizing the same loop to more than one third-party tool/MCP at once, so the "lessons learned" have to be more targeted, not just accumulated in one file.
- Letting the memory system prune or merge lessons instead of only ever growing, once it gets large enough to need it.
- Applying the same self-reflection loop to a user's own private repo or API, rather than a seeded demo one.
Log in or sign up for Devpost to join the conversation.