Smith.forge

• 

  AO SESSION

• 

  AO SESSION

• 

• 

Inspiration

Syndicate Track 1 (Automated Agent Engineering) asks for agents that get better over time—not a one-shot demo with a pretty UI. Most “agent” hacks show a single successful run. We wanted the opposite: a forge that starts weak on purpose, fails loudly, writes durable lessons, and proves the next generation scores higher.

The spark: treat agent engineering like metalwork. Heat (eval), hammer (mutation), quench (memory), inspect (report card)—then forge again.

What it does

SMITH.forge is a meta-agent forge:

1. Propose a cold-start agent for a goal + eval pack (messy invoices, etc.)
2. Eval against real fixtures with a live LLM (no mocked scores)
3. Classify failures (taxonomy)
4. Reflect in natural language
5. Remember durable lessons in SQLite
6. Mutate architecture / prompts for the next gen
7. Report accuracy, cost, latency, memory growth, trajectory

Verified learning loop (bun run smoke, live TensorMux):

$$\Delta \text{accuracy} = +1.0$$

UI path: /start → /dashboard/forge → Forge once → Forge once again. Gen2 reuses memory; the report card shows the lift.

How we built it

• App: TanStack Start (Vite) + React dashboard (forge, agents, runs)
• Loop engine: propose → eval → taxonomy → reflect → memory → mutate → report
• LLM: OpenAI-compatible routing (TensorMux → AgentRouter → Ollama fallback)
• Tools / research: Tavily, TinyFish
• State: SQLite (workspaces, gens, runs, memories, reflections; owner-scoped)
• Deploy: Vercel (smith-tawny-omega / team aliases)
• Judging session layer: Agent Orchestrator (AO) on desktop—real sessions in the demo video; we never invent session counts

Diagrams for judges live in docs/diagrams/ (Excalidraw + mermaid).

Challenges we ran into

1. Honest evals on serverless — Fixture JSON must ship in the bundle; filesystem readdir under /var/task broke forge until we embedded cases.
2. Auth on Vercel + SQLite — Sessions on ephemeral FS can flap across isolates; we gated routes + owner-scoped workspaces, and kept residual risk explicit (no “unhackable” claims).
3. Cold-start that still learns — Gen1 must be weak enough to fail, then Gen2 must clearly reuse memory. We tuned prompts/contracts so the (\Delta) is visible, not faked.
4. AO is mandatory — Cloud AppImage may not register; desktop AO is the path we show on video. We document attempts instead of fabricating sessions.
5. Submit clock — Ship the learning loop + proof first; polish second.

What we learned

• Track 1 rewards visible learning, not domain chrome.
• Memory + reflection only count if the score moves and the UI shows why.
• Sponsor stacks (TensorMux, Tavily, TinyFish, AO) work best when heal/fallback paths are real.
• Be honest about residual risk: durable DB (e.g. Turso/Postgres) is the next step for rock-solid hosted auth.

What’s next

• Persistent store for sessions/workspaces on Vercel
• More eval packs (GROUNDS-lite) and cost/speed tradeoff charts
• Stronger AO↔forge operator playbooks for judges

Built With

• agent
• agentrouter
• agents
• api
• bun
• eval
• excalidraw
• llm
• markdown
• ollama
• openai
• orchestrator
• react
• self-reflection
• sqlite
• start
• tanstack
• tavily
• tensormux
• tinyfish
• typescript
• vercel
• vite
• zod