Foundry

Foundry gives enterprises continuous AI agent evaluation at scale—deterministic checks on 100% of traffic, targeted LLM analysis, automated fixes, data validation and human approval before deployment

Comment

Foundry — Enterprise AI Solution Submission

Inspiration

Every team shipping a customer-facing AI agent hits the same wall: there's no reliable way to know if it's actually behaving well, without either sampling conversations by hand — which doesn't scale and misses systemic issues — or pointing another LLM at every single conversation to "judge" it, which is expensive, harder to audit, and raises an uncomfortable question in any enterprise security review: who evaluates the evaluator?

The idea that shaped Foundry: a lot of what a rule says is mechanically checkable and doesn't need a model's opinion — it needs a compiler. But not everything is mechanically checkable; plenty of failure modes are genuinely contextual and need a model's judgment. So rather than treating "deterministic" and "LLM-based" as competing philosophies, we designed Foundry to let them complement each other — compiled checks carry what they can, and LLM-based analysis fills in the rest, on an agent's own historical conversation data, offline.

What it does

Foundry evaluates an existing AI agent offline, against its own history — not live production traffic. You onboard the agent (via a hosted Platform View, or an MCP connection scoped to the customer's own stack), point it at an exported batch of conversations or a connected log store, and read its manifest: the system prompt as numbered rules, tool schemas, and model config.

Every trace in that dataset runs through a first-pass screen combining deterministic checks — schema validation, regex, local embedding and NLI models — with LLM-assisted analysis where deterministic methods alone can't tell the whole story. Exactly how much of that first pass should be pure code versus model-assisted is still an open design question we're actively tuning, not a settled ratio — the goal is to keep the bulk of screening cheap and auditable while not pretending every failure mode is mechanically checkable.

Findings are aggregated into patterns, and a gate looks at aggregate signal — not individual conversations — before flagging anything as systemic, so a single one-off blip doesn't trigger a full investigation. For patterns that clear the gate, Foundry classifies the likely root cause (routing table first, LLM for the residual — and non-prompt causes exit immediately to an engineering ticket rather than forcing a prompt fix onto a tool bug), generates a structured fix — a real diff to a rule or tool description, never free text — and verifies it by replaying real conversations from the same dataset through the patched manifest, reporting results across three sets: diagnose (traces used to build the fix — expect improvement, discount it), holdout (traces the fix never saw — the number that counts), and guard (previously-passing conversations that must not regress). Nothing ships without a human approving it.

Beyond single-rule fixes, an Architecture Lab surfaces narrative recommendations for bigger orchestration changes — clearly marked as unverified, since a transcript alone can't expose orchestration code the way it exposes a prompt. Once a human implements one and re-uploads the manifest, the same replay-based verification runs on it, including honest cost and latency trade-offs, not just a compliance number.

An admin layer draws a hard line between what's tunable — thresholds, model roles, budgets, rubric weights — and what's invariant: the verification gate, mandatory human approval, bounded retries, DSL-only predicates. Every finding carries full provenance, and every config change is audit-logged.

How we built it

We started from a written architecture spec that had already been through several rounds of critique, resolving gaps like "no verification stage between suggestion and shipping" and "no aggregate boundary on the gate." The build discipline was to make the UI prove those resolutions rather than just describe them — showing all three verification sets even when one is more flattering than the others, and visually separating the verified Fix & Verification flow from the narrative-only Architecture Lab so the two are never confused.

We iterated the design screen by screen against that spec, componentizing the shared shell so a change in one place propagated everywhere, and built a cost/coverage view that makes the deterministic-plus-LLM balance visible and adjustable rather than a hidden implementation detail — because we haven't locked that ratio down yet, and neither should the interface.

Challenges we ran into

The hardest problem was avoiding circularity — an LLM judging an LLM invites a self-preference critique. We handled the parts we could make deterministic architecturally, and for the parts that still need a model, we made sure the judge role and the agent role are never allowed to be the same model, so at least that specific bias is structurally ruled out rather than merely discouraged.

A subtler challenge: Foundry is itself an agentic pipeline with generation, retries, and loops, which means it can commit the exact failures it's built to catch in others. We had to explicitly guard against Foundry's own ratchet growth (a fix generator that only ever adds rules), oscillation across cycles, and a predicate silently returning false forever without anyone noticing it stopped meaning anything.

The most repeated challenge was scope discipline — resisting the pull to let the tool claim more certainty than it actually has, especially around Architecture Lab, where it would have been easy to present a narrative recommendation with the same visual confidence as a verified diff.

Accomplishments that we're proud of

  • A design that treats deterministic and LLM-based checks as complementary rather than competing — using compiled checks where they're sufficient, and reserving model judgment for what actually needs it, instead of defaulting to "run an LLM over everything."
  • A working demonstration of a real contradiction pattern (two rules that can't both be followed) getting confirmed two independent ways — by a local model reading the rule text, and by evidence pulled from real conversations — before any fix is proposed.
  • A generated fix that gets verified the same way it will be judged in practice: replayed on data it never saw, with the result reported honestly alongside the set that was used to build it, so overfitting is visible rather than hidden.
  • A UI that reports plainly when nothing changed, or when a trade-off cuts both ways, instead of presenting every result as an unqualified win.
  • Extending verification past prompts: a human-built architecture change gets put through the same replay-based proof as a one-line rule edit, once it's re-uploaded — closing the gap between "recommended" and "verified."

What we learned

Deterministic and LLM-based methods aren't a hierarchy where one is the "real" answer and the other is a fallback — they're complementary. Classical methods keep the mechanically-checkable majority of a rule set cheap, fast, and auditable; LLM judgment covers what's genuinely contextual and can't be reduced to a pattern match. Where exactly to draw that line is a live design question for us, and we'd rather say so than pretend it's already settled.

Aggregate-vs-per-trace turned out to be an architectural decision, not a nuance — gating on individual conversations would produce constant alert fatigue and no clear read on what's actually systemic.

Showing the gap between a "traces used to build the fix" result and a "traces the fix never saw" result is itself valuable signal — hiding it would make any result look better, and would make the tool less trustworthy for exactly the audience it's trying to earn trust with.

What's next for Foundry

  • Settle the deterministic/LLM balance for the first-pass screen with real usage data, rather than guessing the ratio up front.
  • Build the real MCP adapter so agents that can't share logs off-platform can still be evaluated against their own history without their data ever leaving their stack.
  • Expand the classical-methods toolkit already scoped but not yet built — use-case drift clustering, residual unclassified-failure grouping.
  • Let an Architecture Lab recommendation provision its own sandboxed trial directly, so "a human builds it, then re-uploads" can happen inside Foundry instead of requiring a manual round-trip.
  • Cross-agent pattern detection for fleets — the same contradiction showing up across many agents evaluated by the same team is a distinct, higher-value finding than any single agent's report.

Built With

Share this project:

Updates

Submission history