Tether — Pre-Merge Production Impact Forecasting

Tether ties every MR to live Cloud Monitoring data, traces the blast radius with real BFS/DFS, and forecasts production impact pre-merge -posted on the MR 6 Gemini 3.1 agents ·5 engines · 55 tests

Comment

Tether — Pre-Merge Production Impact Forecasting

Every other merge-request agent reads your code. Tether reads your production.

🔗 Live: https://tether-landing-134108632555.us-central1.run.app 📦 Code: https://gitlab.com/tether7201023/tether 🔴 Dangerous MR (CRITICAL): https://gitlab.com/tether7201023/tether/-/merge_requests/1 🟢 Same change, safe (CONTAINED): https://gitlab.com/tether7201023/tether/-/merge_requests/2

Inspiration

Here's a story every on-call engineer knows by heart.

A merge request looks clean. The diff is small — orders-service now calls analytics-service to log an event when an order is created. The code is correct. The tests pass. Two reviewers approve it. It merges on a Tuesday afternoon.

At 8 PM, traffic peaks. analytics-service is running slow that evening — nothing alarming, P99 around a second. But that new call is synchronous: every order request now blocks, holding a worker until analytics responds. At peak throughput the orders worker pool saturates. Orders start queuing, then timing out. Checkout, which depends on orders, follows it down. Payments too. A one-line, correct, well-reviewed change takes down the storefront.

Nobody did anything wrong. The reviewers saw the diff. What they couldn't see was production — how analytics was actually behaving, how many workers orders really had, and how a stall would ripple through the dependency graph. That information existed. It was sitting in Cloud Monitoring the whole time. It just wasn't in the room where the merge decision was made.

That's the gap. Code review happens in a production-blind vacuum. CI tests run in isolation. APM dashboards show you production — but after you ship, in a different tab, disconnected from the diff. No tool brings live production reality to the moment of the merge.

So we built Tether.

What it does

Tether is a six-agent system on Gemini 3.1 (ADK) and the GitLab MCP server that reviews a merge request against the live production state of every service it touches — and posts a grounded production-impact forecast on the MR, before anyone merges.

Three things make it different from every other code-review agent:

1. It reads live production, not just code

When an MR opens, Tether pulls the real-time state of each touched service from Google Cloud Monitoring — P99 latency, throughput, CPU, memory, observed concurrency, error rate. The forecast is grounded in what production is actually doing right now, not in static analysis of the diff. In our demo, Tether reads analytics-service at a measured P99 of 926 ms and feeds that real number into its capacity model.

2. It reaches opposite verdicts on the same change — because it reasons about runtime

This is the core insight. Whether a change is dangerous lives in how it's called, not in the code.

  • orders --sync--> analytics: orders blocks waiting for analytics. If analytics stalls, orders stalls. The failure propagates upstream.
  • orders --async--> analytics: orders enqueues and moves on. If analytics stalls, orders is fine. The async edge is a firebreak.

So the same change — adding an analytics call to orders — is dangerous done synchronously and safe done asynchronously. We proved this on two real merge requests with identical telemetry:

MR !1 (sync) MR !2 (async)
analytics P99 926 ms 926 ms (identical)
Verdict 🔴 CRITICAL 🟢 CONTAINED
Blast radius 3 services, 2 hops: orders → checkout → payments 0 — async firebreak shields the chain

No linter, no static analyzer, no APM dashboard can tell these two apart, because the difference isn't in the code — it's in how production behaves at runtime. That contrast is Tether's reason to exist.

3. It refuses to make a number up

Every finding Tether posts is tagged measured, estimate, or forecast. A metric it cannot read comes back UNAVAILABLE — never a fabricated value. The capacity math states its assumptions inline. The failure-mode diagram is explicitly a forecast grounded in the capacity model, not an executed test. An SRE trusts a tool that says "I couldn't measure this" far more than one that always has a confident answer — so we made honesty a hard constraint, and it cost us features and bought us credibility.

The pipeline

Six specialized agents run in sequence, each with a distinct job, a focused toolset, and a typed hand-off to the next:

Stage Agent What it does
1 Change Analyst Reads the diff via GitLab MCP; classifies each new dependency sync vs async; scans for CI-weakening changes (see below)
2 Production Reader Pulls live Cloud Monitoring telemetry for every touched service; marks each field OBSERVED or UNAVAILABLE
3 Blast Radius Analyst Reverse-BFS latency blast radius over sync edges (async = firebreak), forward-BFS load, DFS sync-cycle detection — over the real service topology
4 Historian Reads the project's own past incidents and commits to ground the forecast in this team's history
5 Risk Synthesizer Runs the detector registry over change + live snapshots + blast radius; produces ranked, confidence-labelled findings
6 Action The only agent with write access: posts the report, opens line-anchored threads on the exact risky lines, files issues for services it couldn't measure

The CI-integrity guard

Agent-written code is flooding review queues everywhere, and one of the most insidious failure modes is a change that quietly weakens the safety net itself. So Tether's Change Analyst also scans every diff for CI tampering: removed or skipped tests, || true swallowing failures, allow_failure: true, lowered coverage thresholds, narrowed pipeline triggers. A green check that shouldn't be trusted is flagged before it merges. This directly addresses the emerging problem of automated PRs that pass review while silently disarming the pipeline.

The emergent good we didn't plan

If a service Tether needs to assess comes back UNAVAILABLE, the Action agent files a GitLab issue: "Observability gap: <service> is not measurable in production." Reviewing a diff leaves the repository better instrumented than Tether found it. That wasn't in the spec — it fell out of taking the honesty contract seriously. On a later run, the Historian reads those issues back and notes the gaps were since resolved: the system observing its own past actions as institutional memory.

How we built it

A six-agent pipeline on the Gemini Enterprise Agent Platform (ADK), running on Gemini 3.1 Pro via Vertex AI, orchestrating five deterministic Cloud Run engines and the GitLab MCP server.

Architecture

GitLab MR webhook ─▶ Ingestion (Cloud Run)
                          │ launches
                          ▼
              Tether pipeline (Gemini 3.1, ADK SequentialAgent)
  ┌────────────┬────────────┬────────────┬────────────┬────────────┬──────────┐
  ▼            ▼            ▼            ▼            ▼            ▼
Change      Production    Blast        Historian    Risk         Action
Analyst     Reader        Radius                    Synthesizer
  │            │            │            │            │            │
  ▼            ▼            ▼            ▼            ▼            ▼
GitLab MCP  reader-      graph-       GitLab MCP   detector-    GitLab MCP
(diff,      engine       engine       (issues,     registry     (notes,
 source)    (live        (BFS/DFS)    commits)     +math,       threads,
            metrics)                               forecast)    issues)

The division of labor that matters

The LLM agents orchestrate; the engines compute. Little's Law, the BFS/DFS graph traversal, and the live-metrics read are not done by the language model — they are real, tested, deterministic Cloud Run services. Gemini decides when to call each engine and how to interpret the result, but the numbers come from code, not from a model's guess. A judge can read the engine source and the 55 tests and verify it.

The five engines (all live on Cloud Run)

  • Reader engine — reads Cloud Run metrics from Cloud Monitoring into a typed snapshot with per-field OBSERVED/UNAVAILABLE provenance.
  • Graph engine — the algorithmic core: reverse-BFS latency blast radius, forward-BFS load, three-color DFS sync-cycle detection, async-firebreak logic. 16 edge-case tests.
  • Math engine — Little's Law (required_concurrency = throughput × latency) with a labelled first-order saturation estimate. 8 tests.
  • Detector registry — six pluggable detectors (capacity saturation, resource pressure, error-rate amplification, concurrency headroom, blast radius, async-safe pattern), each a function of (change, live snapshot), rendering the MR-ready report. 12 tests.
  • Forecast engine — the honest predicted-failure-mode Mermaid diagram, driven entirely by the math engine's real outputs.

Plus a diff analyzer (10 tests) and a CI-integrity analyzer (9 tests). 55 passing tests total, run by the repo's GitLab CI on every push.

Safety by construction

Only the Action agent holds write tools. The five sensing agents physically cannot modify GitLab — separating reading production from acting on it is a real safety property, not a comment in the code.

The swappable MCP

Tether's GitLab MCP endpoint is a single config value. It runs against a GitLab MCP server over Streamable HTTP using the standard MCP protocol. Because ADK attaches any MCP server identically, the official GitLab MCP (OAuth) and a self-hosted server (token auth) are one line apart. An autonomous, webhook-driven agent can't complete an interactive browser OAuth, so the runtime uses token auth while interactive use can leverage the official server — a deliberate auth-model distinction, not a workaround.

Challenges we ran into

The platform changed under us, mid-build. gemini-3-pro-preview was discontinued in favor of gemini-3.1-pro-preview, and we learned the hard way that 3.1 Pro only serves on the global endpoint, not regional ones. That single misconfiguration silently breaks everything.

The GitLab MCP authenticates with OAuth, not a token. Our first attempts to hit the official MCP endpoint with a Personal Access Token returned 404 — the official server uses OAuth 2.0 Dynamic Client Registration, and a browser callback can't loop back to a headless Cloud Run runtime. That realization shaped our dual-MCP architecture.

Cloud Monitoring is honest, which is inconvenient. Metrics only exist if traffic flowed, and they aggregate over a 1–3 minute lag. An idle service returns UNAVAILABLE for everything — exactly as it should. We built the demo around this truth (driving real load, then forecasting at target enterprise throughput) rather than faking numbers.

Cloud Run autoscales away from saturation. Under load, Cloud Run adds instances rather than letting one instance's worker pool saturate — the opposite of the failure we forecast. We pin the demo service to one instance with a real concurrency cap so the saturation we predict is the saturation you can observe.

Killing our own fabrications. An earlier draft of the math engine hardcoded a time_to_exhaustion constant and an agent that claimed "I tested this." We tore both out. Every number Tether reports is now either a real function of measured input or a clearly-labelled estimate. That discipline is the foundation of the honesty contract.

Accomplishments that we're proud of

  • Live production telemetry at the moment of merge — the one thing no other MR agent does. We verified the Reader returning real data: analytics P99 926 ms, real observed concurrency 11, every field OBSERVED.
  • The same change, two verdicts — 🔴 CRITICAL on the synchronous call and 🟢 CONTAINED on the identical call made async, posted on two real MRs with identical telemetry.
  • A real multi-hop blast radius — BFS/DFS over a real dependency graph, tracing a stall from analytics through orders to checkout and payments (3 services, 2 hops), with the async-firebreak subtlety most impact tools get wrong, proven by 16 edge-case tests.
  • A CI-integrity guard that catches merge requests quietly disarming the pipeline.
  • An emergent good — filing observability-gap issues for unmeasurable services, then reading them back as memory on later runs.
  • Ruthless honesty — measured / estimate / forecast labels on every finding; UNAVAILABLE instead of invented numbers.
  • 55 passing tests, a green CI pipeline, one-command infrastructure-as-code, and a hosted landing page — it ran end-to-end against real MRs, posting line-anchored forecasts and filing issues.

What we learned

  • The bottleneck in shipping isn't writing code — it's knowing what the code will do in production. Agents that generate code are everywhere. The hard, unsolved problem is connecting the merge decision to production reality. That's where the value is now.
  • Honesty is a feature, not a constraint. Refusing to fabricate made Tether more persuasive, not less.
  • How a dependency is called matters more than whether. The sync/async distinction is invisible to static analysis and decisive at runtime.
  • Separate sensing from acting. Giving only one agent write access isn't safety theater — it made the whole system easier to reason about and trust.
  • Let the engines compute and the model orchestrate. The credibility of the forecast comes from real, tested code producing the numbers — the LLM decides what to run and how to read it.

What's next for Tether

  • Learned topology — infer the full service dependency graph automatically from distributed traces, instead of from the diff plus declared config.
  • Historical correlation — when Tether forecasts a risk, cite a matching past incident: "this resembles the outage in issue #7."
  • Auto-resolve threads — when a follow-up push converts a flagged synchronous call to async, Tether resolves its own thread.
  • Multi-cloud telemetry — read from Datadog, Prometheus, and CloudWatch, not just Google Cloud Monitoring.
  • Confidence intervals on forecasts — move from first-order estimates to ranges with stated assumptions.

Built with

Gemini 3.1 Pro · Google Agent Development Kit (ADK) · Vertex AI · Google Cloud Run · Google Cloud Monitoring · GitLab MCP server · Python · FastAPI · MIT licensed

Built With

  • adk
  • bfs
  • cloud-monitoring
  • cloud-run
  • dfs
  • fastapi
  • gemni-3.1-pro
  • gitlab
  • gitlab-mcp
  • google-cloud
  • littles-law
  • mermaid.js
  • model-context-protocol
  • python
Share this project:

Updates

Submission history