AgentForge

Watch AI agents design, execute, & improve in real time. Submit a task; see a meta-agent decompose it into specialized sub-agents, generate evaluation rubrics, & drive self-improvement loops

Comment

AgentForge — Project Story

Inspiration

Multi-agent AI systems are everywhere in the discourse right now — but they're invisible. You read about orchestration patterns, self-improving loops, and agent decomposition in blog posts and papers, but there's no way to watch one work. The outputs are visible; the thinking, evaluation, and evolution are black boxes.

I was inspired by three ideas that converged:

  • Yohei Nakajima's work on self-improving agents — the insight that the improvement loop itself is the interesting artifact, not just the final output
  • VoltAgent's approach to observability — treating execution traces and visual dashboards as native to the agent framework, not bolted on after the fact
  • My own frustration — I've spent the last year building AI tools professionally (including one of the first healthcare MCP servers), and I kept running into the same problem: stakeholders and technical leaders can't evaluate what they can't see

AgentForge exists to make multi-agent systems legible. Not a workflow builder. Not a chatbot. A playground where you submit any task and watch a meta-agent design a team, generate evaluation criteria, orchestrate execution, score the output, and rewrite its own prompts to get better — all in real time.

What It Does

AgentForge takes a vague task prompt and turns it into a fully visible multi-agent workflow:

  1. Submit a task — choose from three demo scenarios (code review, pharmaceutical ingredient extraction, feature request triage) or write your own
  2. Watch the meta-agent decompose it — a node graph shows sub-agents being spawned, each with a generated role and tailored prompt
  3. See agents execute in real time — structured traces show each agent's reasoning, inputs, and outputs as they stream in
  4. Watch self-evaluation happen — a judge agent scores the combined output against a dynamically generated rubric
  5. See the improvement loop — across 2-3 iterations, the meta-agent rewrites sub-agent prompts based on judge feedback, and you can see scores climb in the timeline view
  6. Export the blueprint — after completion, download a reproducible agent architecture spec — every role, prompt, data flow, and evaluation criterion — ready to hand to a developer or paste into a coding agent

The dashboard shows all of this simultaneously: a React Flow node graph of the agent topology, a trace panel with structured execution steps, and a timeline showing score progression with prompt diffs between iterations.

How We Built It

The Process

This project was built using a structured spec-driven development workflow — a plugin-guided process that moves through scoping, PRD, technical spec, and checklist phases before any code is written. Each phase produces a document artifact that feeds the next.

The entire build was planned before the first line of code was written:

  • Scope defined the idea, constraints, and what was explicitly cut (no auth, no persistence, no healthcare domain overlap with my day job)
  • PRD translated that into 12 user stories with acceptance criteria
  • Technical Spec locked in the stack, architecture, data models, SSE event schema, and file structure
  • Checklist broke the spec into 10 sequenced build items, ordered so the riskiest pieces (SSE streaming on Vercel, the orchestration loop) came first

The build itself ran in autonomous mode — each checklist item was dispatched to a coding agent with full spec context. 10 items, scaffold to submission.

The Stack

Layer Technology
Frontend Next.js (App Router) + React + Tailwind CSS
Node Graph React Flow (@xyflow/react)
Backend FastAPI (Python) on Vercel Functions
Agent Framework LangChain
Evaluation Opik (LLM-as-judge)
LLM OpenAI GPT-5.4 Nano
Real-Time Server-Sent Events (SSE)
Deployment Vercel (single project — Next.js + Python)

Architecture

The Python backend runs an orchestration loop: meta-agent analyzes the task → generates an evaluation rubric → decomposes work into 2-4 sub-agents → sub-agents execute via LangChain → a judge agent (powered by Opik) scores the output → the meta-agent synthesizes feedback and rewrites prompts → repeat. Every step emits SSE events that the React frontend consumes through a custom useRunStream hook, updating the node graph, trace panel, and timeline in real time.

A single cheap model (GPT-5.4 Nano) powers every role — meta-agent, sub-agents, and judge. The self-improvement loop multiplies LLM calls ($N$ agents $\times$ $K$ iterations), so cost conservation was a design constraint from the start. A full 3-iteration run with 3 sub-agents uses roughly 50-80K tokens and costs about $\$0.02$.

Challenges We Ran Into

The Integration Seams

The individual pieces — backend orchestration, SSE streaming, React dashboard — all worked in isolation. The pain was in the seams between them:

  • Next.js proxy buffers SSE streams. The dev server's rewrite proxy collects the entire response before forwarding it, which defeats the purpose of streaming. We had to bypass the proxy for SSE connections and hit FastAPI directly during local development.
  • Firefox blocks cross-origin fetch silently. After adding the direct SSE connection, POST /api/run was still going through the proxy — but Firefox's stricter CORS enforcement meant it failed silently. The fix was using relative URLs through the proxy for POST requests while keeping SSE direct.
  • Backend event schemas and frontend expectations drifted. The backend used from/to keys for data flow edges; the frontend expected source/target. Scores showed 8.0 instead of the weighted 7.92 because evaluation_complete was missing the overall_score field. Cost displayed $0.00 because amounts under a penny round to nothing at 2 decimal places.

These are the bugs that unit tests don't catch and code review doesn't find — you only discover them by running the full system end-to-end.

The Admin Auth Saga

The app needed a simple admin bypass for rate limiting and export gating — nothing fancy, just "enter a password, unlock features." This went through three failed approaches:

  1. Client-side hash comparison — broke because the hash wasn't available in the browser environment
  2. NEXT_PUBLIC_ env var string comparison — broke because Next.js env var loading behaves differently in local dev vs. Vercel deployment
  3. Server-side verification endpoint — the approach that actually worked. A POST /api/py/admin/verify endpoint hashes the token server-side and compares against the stored secret. No env vars in the frontend, works everywhere.

The lesson: when a frontend-only approach keeps breaking across environments, move the logic to the backend. It's simpler, it's one source of truth, and it doesn't depend on framework-specific env var loading behavior.

Cost Tracking Was Lying

The cost display showed numbers that were 6x lower than the OpenAI dashboard. Three compounding issues:

  • Pricing constants were set to GPT-4o Mini rates ($\$0.15/\$0.60$ per 1M tokens) even though we'd switched to GPT-5.4 Nano ($\$0.20/\$1.25$ per 1M)
  • Sub-agents were estimating token counts with len(text) // 4 instead of reading real usage_metadata from the API
  • Output token pricing was wrong by a factor of 2

Each error was small. Together they made cost transparency — one of the PRD's explicit requirements — meaningless. Fixed by enabling stream_usage=True on the LLM client and updating all pricing constants.

Accomplishments That We're Proud Of

  • The improvement loop actually works. Scores genuinely climb across iterations — not because we tuned it to, but because the meta-agent's prompt rewrites address real weaknesses identified by the judge. Watching a score go from 6.5 → 7.8 → 8.4 in real time is the demo moment.
  • The blueprint is genuinely useful. The exported agent architecture spec is detailed enough to hand to Claude or another coding agent and reproduce the entire multi-agent system. It's not a summary — it includes every agent's final tuned prompt, the evaluation rubric, data flow connections, and orchestration pattern.
  • Three demo scenarios that showcase different capabilities. Code review (analytical), ingredient extraction (structured data from unstructured text), and feature request triage (prioritization and judgment) — each produces a meaningfully different agent topology.
  • Cost per run is about two cents. The single-model-tier decision (GPT-5.4 Nano for everything) kept costs low enough that the app can be demoed freely without watching a billing dashboard.

What We Learned

Spec-driven development compresses debugging time. The 10-item build went cleanly — no structural revisions, no architectural pivots. The spec had already resolved the hard decisions (SSE vs. WebSockets, tab navigation vs. routing, single model tier). The debugging time was spent on integration issues, not design issues. That's the right kind of debugging.

The gap between "it works" and "it works correctly" is where the real effort lives. The build produced a functional app. Two iteration rounds of hands-on testing found the bugs that mattered: admin auth that accepted any string, cost numbers that were wrong by 6x, rate limiting that locked out local development. Code that passes its own tests isn't the same as code that works for users.

Self-improving agents are more compelling to watch than to read about. The pitch — "agents that get better" — sounds abstract until you see the timeline cards fill in, scores climb, and prompt diffs show exactly what changed. Observability transforms a black box into a story.

What's Next

  • Persistent run history — save past runs to a database so users can revisit, compare, and learn from previous agent architectures
  • Shareable runs — unique URLs for completed runs so users can share results with colleagues
  • Blueprint format options — export as JSON config, architecture docs, or framework-specific specs tailored to different coding agents
  • More demo scenarios — expand beyond three to cover more domains and task types
  • Longer improvement loops — allow 5+ iterations for complex tasks where the improvement curve hasn't plateaued

Built with Next.js, FastAPI, LangChain, Opik, OpenAI, React Flow, and Tailwind CSS. Deployed on Vercel.

Built With

Share this project:

Updates