Hyrum

Workspace overview: evidence capture, drift triage, replay health, and policy status.
A signed evidence receipt records the observed contract and decision context.
Differential replay compares baseline and candidate behavior before merge.
Filter the evidence ledger by tier, consumer, endpoint, severity, and time window.

Inspiration

A schema diff can be clean and a customer can still break.

A client might resend an undocumented next_cursor, depend on response ordering, or branch on an error message that was never part of the contract. Those dependencies usually become visible only after a release fails.

Hyrum looks for evidence of that reliance before the change ships.

What it does

Hyrum builds a per-consumer ledger of API behavior that is missing from the published OpenAPI document. Sensitive data is redacted during capture, before evidence leaves the customer’s boundary.

It separates observations by how strongly they demonstrate reliance:

T1: A value returned by the API appears in a later request from the same complete session. Policy may block a release on this evidence.
T2: Request behavior changes alongside an observed response pattern. Hyrum raises a warning.
T3: A consumer received the behavior, but reliance is unproven. It remains visible and never blocks.

When an API change is proposed, Hyrum replays the same redacted corpus against the base and candidate builds. Reviewers see the behavioral difference, the affected consumer, and the evidence receipt behind the decision instead of a generic compatibility score.

How we built it

The capture agent and replay runner are written in Go and use the same versioned extractor package. This was an important constraint: capture and replay cannot disagree about what counts as evidence.

A Fastify control plane validates ingestion and stores tenant-scoped records in PostgreSQL. The review workspace is built with Next.js and React, with WorkOS handling human authentication. Replay reports identify immutable container images and are signed with Ed25519 so the control plane can verify them before recomputing a gate decision.

OpenAI’s Responses API and GPT-5.6 help name behaviors, explain likely impact, prioritize replay work, and draft notices. That work is advisory. Model output is schema-validated and cannot create evidence, promote an observation to T1, alter replay results, or decide whether a release is blocked.

We used OpenAI Codex throughout the project for implementation, testing, product decisions, and deployment work.

Challenges we ran into

The main challenge was distinguishing observation from reliance. Recording every undocumented field would have produced a large, noisy inventory. Hyrum needed evidence that a specific consumer changed its behavior because of something the API returned.

Privacy added another constraint. The system still needs enough context to correlate a response with a later request, but it should not retain request bodies, credentials, cookies, signed URLs, or other sensitive values. Redaction therefore had to happen during capture rather than later in the control plane.

Replay also needed to be reproducible. Timestamps, generated identifiers, and other nondeterministic values can make two equivalent responses look different. Hyrum masks those values and repeats changed cases before treating a difference as stable.

Accomplishments that we’re proud of

We completed the full path from captured traffic to a reviewable release decision:

Capture and redact an API session.
Extract consumer-specific evidence.
Store a signed receipt.
Replay base and candidate builds.
Verify the replay report.
Recompute the release gate from deterministic evidence.

The acceptance scenario processes 50,000 scripted events and exercises T1, T2, and T3 behavior. That is a deterministic test corpus, not a production-traffic claim.

We also deployed a working review experience where a developer can move from an aggregate gate result to the exact receipt responsible for it.

What we learned

Seeing undocumented behavior is easy. Proving that a consumer relies on it is the useful part.

We also learned that AI is most valuable here when it helps a reviewer understand evidence, not when it replaces the evidence. Keeping extraction and gate decisions deterministic made the GPT-powered explanations more useful because their authority was deliberately limited.

Finally, sharing one extractor between capture and replay removed an entire class of disagreements that separate implementations would have introduced.

What’s next

The next step is to run Hyrum with real API teams and measure where its evidence rules produce useful warnings or false positives. We also want to make local replay setup easier and improve the path from a receipt to a focused regression test.

Broader CI and notification integrations should follow that validation rather than getting ahead of it.

Try it out

Open hyrum.vercel.app. The landing page is public.

Select Open workspace and sign in with WorkOS to enter the tenant-scoped review workspace. Follow the workflow through Capture → Ledger → Change gate → Assertions, then open Replay integrity to inspect the signed base-versus-candidate comparison.

Open a T1 receipt to see the evidence behind a blocking decision, then compare it with the T2 warnings and T3 exposure records.