SafeTelemetry

Your app is fine. Your alerts went silent. SafeTelemetry catches the Collector change that silences them before you merge, and a GitLab Duo agent opens the fix.

Comment

Path A: Start Fresh · Supervised. The agent opens draft merge requests; a human merges.

Inspiration

One config line in an OpenTelemetry Collector can rename every metric your alerts read. Nothing errors. The dashboard shows "No data", and the paging alert simply never fires again.

This is already scheduled. The spanmetrics connector README says: "After one release cycle, the unit will switch to s." When it does, *_duration_milliseconds_* becomes *_duration_seconds_*. I searched public code: 162 public repositories, 1,011 dashboard panels, and 37 alerts query the old names (lower bounds). It has happened before: after the connector renamed latency to duration, the official OpenTelemetry Demo's own dashboard showed no data (opentelemetry-demo#849). In March 2026 a Grafana upgrade renamed its own alerting metrics with no changelog entry, and existing alert rules broke (grafana/grafana#120761).

It matters more as teams hand incidents to AI SREs. The ones I looked at (AWS DevOps Agent, Azure SRE Agent, Datadog Bits AI SRE, HolmesGPT) wake up on an alert and check their guesses with metric queries. Kill the metric, and the agent never wakes up, or it reads the empty result as healthy.

What it does

  • Contract gate on every merge request. CI runs the base and the candidate Collector on the same synthetic telemetry, then checks every consumer: alert rules, dashboards, and the PromQL that AI agents run. It reports exactly what loses data and why: "1 alert(s), 1 agent query(ies) and 4 dashboard panel(s) lose data. Trigger: feature gate connector.spanmetrics.useSecondAsDefaultMetricsUnit turned on."
  • Does the alert still fire? promtool replays the candidate's real output into the original alert rules. This catches changes that keep names but break labels.
  • Does "jump to trace" still work? Turning off exemplars or the OpenMetrics format keeps every name and label, so the panel still draws and the trace link silently opens nothing. The gate warns when a link that worked on the base breaks on the candidate.
  • AI apps break the same way. The OpenTelemetry GenAI semantic conventions are still in development. v1.37.0 renamed gen_ai.system to gen_ai.provider.name. When the app's instrumentation moves to 1.37, the token cost alert goes silent and every provider merges into one line that still looks fine. The gate judges an instrumentation upgrade (app/telemetry.env) like a Collector change: "2 alert(s), 1 agent query(ies) and 2 dashboard panel(s) lose data. Trigger: app GenAI semantic conventions 1.36.0 → 1.37.0."
  • Severity tiers. A lost alert, agent query, or recording rule fails the pipeline, and "Pipelines must succeed" is on, so the merge is blocked. A panel-only loss is a warning.
  • A GitLab Duo flow fixes it. On a failed pipeline, a triage step decides fix or skip. A fixer reads the merge request's intent: if the goal is seconds, it moves the alert and agent queries to *_seconds_* and divides thresholds by 1000; otherwise it pins histogram: {unit: ms} in the Collector. It opens a draft merge request and copies the gate's impact table into a comment.
  • The agent cannot grade its own work. The same gate runs on the fix. agent-guard fails the fix if it touches anything outside the Collector and consumer folders, changes more than 3 files or 40 lines, deletes an alert, lowers its severity, makes it wait longer, or deletes an agent query. Expected alerts live in a human-owned file, and CODEOWNERS names the human owner.
  • Forecast mode that acts. Every Monday a scheduled pipeline turns on each upcoming Collector feature gate, one at a time. On Collector 0.162.0, 3 of 5 would silence my alerts. The first run opened one issue per gate (#2–#4) and draft !22, which pins histogram: {unit: ms} before the unit default flips. After that it stays quiet until something new breaks.
  • The same contract on the deployed revision. After a human merges, main deploys the checkout app to Google Cloud Run with the Collector as a sidecar. The new revision gets 0% traffic, the app's real spans go through its Collector, and the same gate judges that output against the live revision. Only a pass is promoted; a manual job rolls back.

How I built it

  • GitLab CI/CD: unit-tests, contract-gate (JUnit report in the MR widget), agent-guard, SAST and secret detection, a scheduled forecast + forecast-handoff, a scheduled eval, and Cloud Run build-app → deploy-candidate → smoke-candidate → promote (+ manual rollback).
  • Google Cloud Run: two containers per revision (the app as ingress, the official Collector image pinned by digest as a sidecar), tagged 0%-traffic revisions, keyless CI auth through Workload Identity Federation limited to this project's protected main branch.
  • Official otelcol-contrib, mimirtool, and promtool releases, verified by SHA-256 before running.
  • Python for the gate: mimirtool extracts the metrics each consumer references; set comparisons decide what broke; rules fill the cause and suggested-fix columns. No LLM decides pass or fail.
  • GitLab Duo Agent Platform custom flow, triggered by Pipeline events (Failed): triage, then a fixer for merge requests or a planner for forecasts.
  • AGENTS.md tells agents what they may edit; CI enforces it.

Challenges I ran into

  • My agent only knew one kind of break. The first agent eval scored 3/5: it stopped on a lost label, and triage skipped a Collector that would not start. Adding a rule per case would have fit the test set, so I rewrote the flow around one contract instead: every consumer keeps getting the data it reads. All five then passed on the new version.
  • The first Collector output lies. Counters read 0 on the first scrape, so the gate waits until the output is stable for 20 seconds.
  • My first eval missed two panels. Two error panels filtered on a label the change removed. The names still matched, so the name check passed them. I added a label-filter rule and reran with the labels unchanged: 14/14.
  • Pinning tool arguments broke writes. Fixing project_id in the flow's tool options made every write call return 404 on GitLab.com. Removing it fixed the flow.
  • Cloud Run gives a service one port; the Collector needs two. The app is the ingress container and sends its spans to a Collector sidecar on localhost, and a token-guarded route returns the sidecar's output to the deploy check.
  • My first deploy check saw one span. Cloud Run adds a traceparent header and marks most requests unsampled, so the SDK's parent-based sampler dropped every span after the first. The gate failed that deploy, and I reproduced the drop locally before switching the app to always sample.

Accomplishments that I'm proud of

  • Gate eval: 25/25. 25 real changes (Collector configs and the app's GenAI instrumentation), labeled from upstream docs before the gate ran. First 14: 12/14 fully correct, then 14/14 after one rule fix with the labels unchanged. Then 7 trace-link and processor cases: 7/7. Then 4 GenAI cases: 4/4.
  • Agent eval: 5 scenario merge requests, expected fix side labeled before any run. First flow version 3/5; after the rewrite, 5/5, every fix passing the gate and agent-guard. Then 3 held-out scenarios: 2/3. The miss took the right side but kept an old label value, and the gate kept its pipeline red.
  • Same break, two intents: MR !3 turns on the seconds unit with no stated goal, so the agent pins ms in draft !15, which a human merged. MR !4 says "standardize on seconds", so the agent moves the alert and agent query to seconds (500 → 0.5) in draft !13.
  • Speed (5 scenarios): draft fix a median of 1.1 minutes after the gate went red, green a median of 4.1 minutes after.
  • 94 tests, including a 5 consumer × 7 change matrix that runs the real functions.

What I learned

  • Rules should decide and agents should propose. The gate is deterministic and free to rerun; the agent only does what rules cannot, like reading a merge request's intent.
  • An agent's fix must be checked by something it cannot edit.
  • Observability changes need tests like code changes. The test is "does every consumer still get data", not "does the Collector start".

What's next for SafeTelemetry

  • A contract lookup for agents: an MCP tool an AI SRE can call before it trusts a query.
  • Check the Collector's own otelcol_* metrics, which also change between releases.
  • Log → trace links, and more receivers and processors in the synthetic input.

Try it

  1. MR !3 and MR !4: both pipelines fail at contract-gate; the job log and the MR test widget show the impact table.
  2. Draft MR !15 (merged) and draft MR !13: the agent's two fixes, both green including agent-guard.
  3. Issues #2–#4 and draft MR !22: the forecast planner's output.
  4. Locally: pip install -r requirements-dev.txt, python -m pytest tests/, GATE_BASE_REF=HEAD~1 python gate/run.py, python evals/run.py.
  5. Live app: https://safetelemetry-checkout-926147183407.us-central1.run.app — it names the revision and the pipeline that deployed it.

Built With

  • gitlab
  • gitlab-ci
  • gitlab-duo
  • google-cloud-run
  • grafana
  • mimirtool
  • opentelemetry
  • opentelemetry-collector
  • prometheus
  • promtool
  • python
Share this project:

Updates

Submission history