Causeway

• 

  Five scenarios: two locked, three adversarial. Frontend renders instantly, falls back to a labelled replay if the backend is cold

• 

  57,000 log events replayed over WebSocket with real progressive status: connecting → replaying → detected → correlating → complete

• 

  Onset is found from a log template never seen before — not chosen from the deploy list — so the temporal term stays unbiased.

• 

  Ranking, per-term breakdown, and verified claims in one view. B (blast radius) is shown greyed — measured, changed nothing, dropped.

• 

  Every term expands into the arithmetic behind it. E=0.936 is a two-proportion z-test you can check by hand — nothing's a black box.

• 

  An injected false claim, added to test the checker. The verifier catches it and shows why. Refuted claims are shown, never dropped.

• 

  The model investigates second and can only conclude via typed claims from a closed list — each checked against telemetry first.

• 

  The score is a ranking, not a confidence — an innocent deploy hit 0.655, a real cause hit 0.003. No threshold separates them

• 

  The distractor is more recent (T=0.951 vs 0.741) but E=0.000 — its errors never moved. The engine still ranks the real cause first.

• 

  Three scenarios the engine is expected to fail. Prediction recorded before the run. This one misses by 0.050, exactly as predicted.

## Inspiration

Most "AI for incidents" tools ask an LLM to look at some logs and guess what broke. That's fast, but it's also just a confident-sounding guess — there's no way to check whether the model actually reasoned about the evidence or just pattern-matched to "most recent deploy = culprit."

Causeway inverts that order. A deterministic correlation engine ranks candidate causes first, using an explicit, inspectable formula. Only then does an AI investigator agent run — and it's constrained to make claims that are checked against the underlying telemetry before they're ever shown.

What it does

Causeway ingests synthetic telemetry (logs, metrics, deployments) for an incident, detects the incident, and ranks every deployment in the lookback window as a candidate cause using a three-term score: Score = 0.353T + 0.353E + 0.294*M

• T — temporal proximity to incident onset
• E — how strongly the deploy's service correlates with the error-rate spike
• M — magnitude of metric deviation from baseline

Only then does a tool-calling AI agent (Gemini) investigate. It queries the same telemetry the engine used, and every claim it makes — "this deploy preceded onset," "this candidate is ruled out" — is a typed object checked against real data before it's rendered. Claims come back VERIFIED, REFUTED, or UNVERIFIABLE. Nothing is ever shown that wasn't checked.

The demo scenario: a payments deploy causes an incident, while a search service deploy lands even closer in time (30s before onset vs. 180s) and looks equally suspicious. The engine still ranks payments first, because its error rate actually spiked and search's didn't. The agent independently verifies this and explicitly rules search out with real evidence — it isn't told the answer, it checks.

How we built it

• Correlation engine: pure Python, no ML — an explicit, inspectable formula over telemetry queried from SQLite
• Claim verification: a closed enum of claim kinds (deploy_precedes_onset, error_rate_change, metric_deviation, candidate_ruled_out, etc.), each with exactly one verifier function. The agent can only emit claims from this enum — a claim kind with no verifier literally cannot be produced.
• Agent: Gemini 2.5 Flash via a provider-adapter layer, so the transport is swappable without touching the claim schema, verifiers, or agent loop.
• Scenario generator: parameterized, seeded, and labeled — every generated scenario carries ground truth, which is what makes the evaluation honest.
• Frontend: React + WebSockets, with a cold-start fallback to a pre-recorded replay (clearly labeled as such) if the live socket doesn't connect in time — free-tier hosts spin down, and the demo shouldn't die because of it.
• Deploy: FastAPI on Render, static frontend on Netlify. The AI panel is served from a committed, verified cache of one real Gemini investigation rather than requiring a live API key in production — cheap, zero secret exposure, and honest about being a cached real response rather than a fake one.

Challenges we ran into

The weight-fitting trap. Our first instinct was to fit the scoring weights against our own generated scenarios. That's circular — we inject the incident, so we already know the label, and recovering it proves nothing. We caught this before shipping it: instead, the weights are hand-chosen priors, and we ran a sensitivity sweep across weight space and a required set of adversarial scenarios the engine is expected to fail on. We report accuracy with the adversarial cases broken out separately, rather than blending them into a misleadingly clean number.

Gemini's function-calling schema is weaker than Anthropic's. Our claim contract was originally a discriminated union enforced at the tool-schema level — a malformed claim was literally impossible for the model to emit. Gemini's function declarations don't support the schema features that made that guarantee possible (no anyOf, no additionalProperties, no strict equivalent). We didn't loosen the contract — we moved enforcement into our own code. Under the old setup, a bad claim was impossible to emit. Now, it's possible to emit and impossible to accept. Same end guarantee, different layer, and we think that distinction matters.

A relative path that only worked in development. Our AI cache lookup used a path relative to the working directory. Every test and every manual run happened to be launched from the repo root, so it always worked — until deployed, where the working directory was different, and the cache silently missed. 174 passing tests never caught it, because none of them changed directory before testing. We fixed the path and added a test that does.

Accomplishments that we're proud of

Finding, and then deliberately not shipping, a fit that scored 11 points higher than our chosen weights — because it only won by nearly ignoring when something was deployed. That's the clearest evidence we have that our weights are genuinely defensible priors rather than a number we backed into.

We also caught our own weight-fitting circularity before it shipped, not after: recovering a label we injected ourselves would have proven nothing, so instead we built a required adversarial scenario set and reported accuracy with those cases broken out separately, rather than blending them into a misleadingly clean number.

What we learned

The most interesting result in this project isn't a win — it's a fit we found and deliberately didn't take. A sensitivity sweep showed that a different weight vector (T=0.05, E=0.30, M=0.65) scores 11 points higher than our shipped priors on our own scenario set. We didn't adopt it, because it only wins by nearly ignoring when something was deployed — a formula that does that would be indefensible against a real incident. The fit exists, it looks better on paper, and taking it would have been the wrong move. That's the clearest evidence we have that the weights are genuinely priors and not a number we backed into.

What's next for Causeway

• Extend the claim schema to support cross-service causal chains, not just single-deploy attribution
• Add a real observability integration (behind a flag) so the engine can run against live telemetry, not just synthetic replays
• Expand the adversarial scenario set and publish the full accuracy breakdown as an ongoing benchmark rather than a one-time number

Built With

• fastapi
• gemini-api
• google-generative-ai
• javascript
• python
• react
• render
• sqlite
• vite
• websockets