Upstream breaks it. Insurixa fixes it before it ships.

Code fixes what is provably safe at zero model cost. GitLab Duo agents handle only the judgement. All nine DevSecOps stages prove every change, and Google Cloud records every release with zero stored keys. 6 of 6 real upstream files repaired or rightly refused · 93 of 500 credits for the whole month · about 1 g CO~2~e per release pipeline.

Your Helm chart worked last year. Today Kubernetes rejects it. Your pipeline still logs in with a variable GitLab removed two releases ago, and the linter calls it valid. Your latency dashboard went quiet because a metric moved from milliseconds to seconds under a new name. Nobody changed anything. Upstream did.

Insurixa is insurance against breaking changes. It reads what Kubernetes, GitLab CI/CD and OpenTelemetry have removed, fixes what code can prove is safe, hands the rest to GitLab Duo agents, ships every fix through all nine DevSecOps stages, and checks production afterwards. ==No person in the loop, and every step is public.==

Try it Link What you will see
Live playground insurixa.web.app/try paste a manifest, a .gitlab-ci.yml or instrumented code; the real packs run in your browser
Platform guide insurixa.web.app/platform the production architecture on GitLab and Google Cloud, with the live ledger
GitLab project showcase flows, pipelines, issues and merge requests, all public
Production service checkout-production place an order, refund it, watch the emitted names (free tier: the first load can take a minute)
Demo video youtu.be/x8GfZbp2FwU the whole loop in 2 minutes 52 seconds

On this page: Inspiration · Why the AI era needs this · What it does · How we built it · Agent evaluation · Google Cloud · Sustainability · Challenges · What's next · Path B checklist


Inspiration {#inspiration}

Where the idea came from

Insurixa did not start as a hackathon idea. It started with a question I kept running into while building other things: why does working code break when nobody touches it?

The answer, almost every time, was an upstream project retiring something on its own schedule. Kubernetes removes an API version. GitLab removes a CI/CD variable. OpenTelemetry renames a metric and changes its unit. None of these are bugs. They are announced in release notes and deprecation guides that few people read until the upgrade fails.

Before this hackathon I built Attrition (MIT), a dataset of every OpenTelemetry semantic convention across 26 tagged releases, with a verdict for each retired name: renamed, replaced, split, dropped, or dependent on the span kind. Then I asked the obvious follow-up question for a world where AI writes most new code: does AI-written code use names that upstream has already retired?

What I measured

I built the Semconv Drift benchmark[^bench] and graded 17 models in code, not by another model, against those 26 releases:

  • 16 of 17 models wrote retired names, 357 times in total. 83% of those names had been retired for more than a year.
  • 12 of 13 models wrote names they had correctly replaced when asked directly. Knowing the answer is not the same as using it.
  • 7 of 17 "fixed" a metric by converting its unit and keeping its retired name. That breaks it in both directions, and the diff still looks like a fix.

Insurixa platform guide

That result is the reason Insurixa exists. The hackathon's premise is that code generation is solved. If that is true, the expensive part of software moves to everything that happens after the code is written: noticing what upstream changed, deciding what is safe, editing, reviewing, shipping and confirming production. That work does not go away when AI writes the code. It grows, because AI writes yesterday's API with total confidence.

Why it matters beyond one ecosystem

Over the following days I checked whether the same pattern holds outside OpenTelemetry. It does:

  • Kubernetes. Three Prometheus community charts published today still render policy/v1beta1 PodDisruptionBudget or autoscaling/v2beta1 HorizontalPodAutoscaler. I reproduced the failure on a real Kubernetes 1.32 cluster: helm install fails the moment a team turns on a disruption budget or autoscaling, which is exactly when they are going to production.
  • GitLab CI/CD. Across the root pipelines of the 500 most-starred GitLab.com projects, 8 use features GitLab has already removed, and 5 very active ones use retry:when values GitLab removes in 20.0. GitLab's own CI linter passes one broken pipeline that logs in with an empty password, because a removed variable is not a syntax error. It simply expands to nothing.
  • OpenTelemetry. Five popular instrumented repositories contained 107 retired names in live code, 67 of them in Jaeger.

The problem

Finding these is the easy part. Every platform team I can picture already has a scanner that prints warnings. What nobody has is a system that finishes the job: decides, fixes, proves and ships, then keeps watching. That is the life after code that Insurixa takes over.

Why the AI era needs this layer {#ai-era}

It is tempting to think AI makes this problem disappear: if a model can write the code, surely it can update it. The evidence says the opposite, for five reasons.

1. AI multiplies the surface that upstream can break. Every manifest, pipeline and instrumented service an agent writes is another place where a removed API can hide. When writing code becomes nearly free, the amount of code that depends on yesterday's upstream grows faster than any team can review it. The bottleneck moves from writing to keeping things working.

2. A model's knowledge is frozen, and upstream is not. A model learns the ecosystem as it was at its training cutoff \( t_0 \). Upstream keeps removing things after that date, so the gap between what a model writes and what upstream accepts can only widen with time:

$$\text{drift}(t) = \big|\, \text{Removed}(t) \cap \text{Used}(t_0) \,\big|, \qquad \frac{d}{dt}\,\text{drift}(t) \ge 0$$

That is exactly what the benchmark showed: 83% of the retired names models wrote had been retired for more than a year. The model was not careless. It was confidently out of date.

3. Breaking changes in the backend fail silently and late. A removed frontend API usually breaks loudly in a browser. Backend removals do not. A Kubernetes manifest fails only on the cluster that no longer serves it. A removed CI variable expands to an empty string, and GitLab's own linter still calls the pipeline valid. A renamed metric does not error at all: the old dashboard simply stops receiving data. These failures surface on upgrade day, in production, often weeks after the code was merged. That is the most expensive possible moment to find them, and it is precisely the part of the lifecycle that AI code generation never sees.

4. Agents need ground truth, not more confidence. An agent asked to "upgrade this" will produce a plausible diff. Without upstream's own removal data, it cannot know whether policy/v1 changes behaviour for an empty selector, or whether a metric's unit moved. Insurixa gives the agent a sourced verdict and a reason, so it only has to make the judgement code cannot. Then the pipeline proves the result against a running service, a strict schema and production itself.

5. Human review does not scale with AI output. If every AI-written change still needs a person to check it against a migration guide, the person becomes the bottleneck. Insurixa shrinks review to the findings that truly need judgement. In the evidence I collected, code alone could fix most findings:

$$\frac{|F_{\text{safe}}|}{|F|} = \frac{6}{8} = 75\% \ \ (\text{go-redis metrics}), \qquad \frac{|F_{\text{safe}}|}{|F|} = \frac{28}{28} = 100\% \ \ (\text{removed features in the GitLab top 500})$$

Everything code can prove is fixed with zero model calls and zero review. What remains goes to an agent with its reason attached, and then through the same nine-stage pipeline a person's change would face, with an audit trail at the end.

This is why Insurixa is a backend system, not a chatbot. It does not ask anyone to trust an answer. It turns upstream's published facts into code, lets agents act only where judgement is needed, and lets the pipeline and production decide whether the result is correct. In an era where AI writes more code than people can read, that is the layer that keeps software working after it is written.


What it does {#what-it-does}

The loop

Insurixa has three packs, each built from its upstream project's own published removal data: Kubernetes APIs, GitLab CI/CD keywords, variables and templates, and OpenTelemetry names. Around them, Insurixa runs the entire upgrade on GitLab, and no Insurixa step waits for a person.

Architecture

There are three ways a change enters the loop:

  1. Production tells us. A scheduled pipeline exercises the live service, reads what it emits, and opens an issue when anything is retired. The issue starts the Insurixa upgrade flow, which plans, writes the change and opens a merge request that merges itself when every gate passes.
  2. A developer copies something old. When an Insurixa gate fails on anyone's merge request, code commits every provably safe fix first. The Insurixa pipeline repair flow then fixes what needs judgement, on the same branch, and explains what it did.
  3. Production drifts on its own. When monitoring sees production emit something the verified build never emitted, it restores the last good config or rolls back, then opens an incident. The Insurixa incident response flow finds the commit and the line and ships the fix.

A fourth path looks ahead: the upgrade radar checks the pipeline against the next GitLab release and the manifests against the next Kubernetes version, and opens an issue while the old form still works.

All nine stages, each doing real work

Stage What Insurixa does there Jobs
Plan reads production and opens an issue; the radar opens issues for removals one release ahead semconv-watch, upgrade-radar
Create code commits safe fixes; three GitLab Duo flows write the rest insurixa-safe-fix, 3 flows
Verify 17 unit tests, the three packs as a CI/CD component, a telemetry contract that runs the service, kubeconform strict against Kubernetes 1.33, Ruff with Code Quality insurixa-*-scan, telemetry-contract, kubernetes-schema
Secure SAST and secret detection, which feed the policy gate semgrep-sast, secret_detection
Package the service image per commit and a bundle of the Kubernetes manifests; the image the component itself runs is built first in the same pipeline, so it is tested before anyone else uses it build-image, manifest-bundle
Release a GitLab release whose notes tell dashboard owners which names changed and which units moved release-notes
Configure environment config as code, applied under one resource group per environment configure-staging, configure-production
Monitor after every deploy and every 6 hours: heal production first, then open an incident monitor-production
Govern checks insurixa-policy.yml rule by rule, posts the audit where the work started, closes the issue, and writes the pipeline's record to the ledger on Google Cloud governance, ledger

All nine stages, green

Code decides, agents judge

The central design choice is that a model never decides what is safe. Code does, from upstream's own data. For a finding \( f \) with the set of documented replacements \( R_f \):

$$\text{safe}(f) \iff |R_f| = 1 \;\land\; \text{when}(R_f) = \text{always} \;\land\; u_{\text{old}} = u_{\text{new}}$$

One replacement, it applies in every context, and the unit does not change. Safe findings are fixed by code at zero model cost. Everything else goes to an agent, with the reason attached:

Retired Why find-and-replace is wrong What has to happen
Kubernetes autoscaling/v2beta1 autoscaler the target changed shape; the API server rejects a bumped object move targetAverageUtilization: 80 into target: {type: Utilization, averageUtilization: 80}
GitLab CI_JOB_JWT_V2 there is no drop-in variable declare id_tokens: with an audience and read that variable
GitLab retry:when: stuck_or_timeout_failure one reason became four choose which failures this job should really retry on
OpenTelemetry db.client.connections.wait_time the unit changed rename and convert every recorded value: \( v_{\mathrm{s}} = v_{\mathrm{ms}} / 1000 \)
OpenTelemetry net.peer.name the replacement depends on span kind server.address on client spans, client.address on server spans
Kubernetes PodSecurityPolicy no replacement API exists stop, and explain the options to a person

The vocabulary, in one place:

Safe fix : a finding with exactly one documented replacement that always applies and keeps the unit, so code commits it with no model call (SAFE_BUMP, SAFE_RENAME, or a safe OpenTelemetry rename).

Judgement : a finding whose replacement changes shape, unit or meaning (SCHEMA_CHANGE, SPLIT, SPAN_KIND_DEPENDENT), handed to a GitLab Duo flow with its reason.

No replacement : upstream removed something and offers nothing equivalent (NO_REPLACEMENT); the right answer is to stop and explain.

Stale field : a field left behind after an apiVersion bump (STALE_FIELD); the object looks upgraded but the API server rejects it.

Ledger : one public record per main pipeline in Firestore on Google Cloud, written by the pipeline itself.

This is what makes the system both cheap and trustworthy. Agents are not asked to remember migration guides. They are handed a precise, sourced reason and asked only for the judgement that code cannot make.

Seven real runs, every step linked

I played the developer and the teammate. Insurixa played everyone else.

Run What happened Evidence
1 Production emitted 7 retired names. The pipeline opened an issue, the upgrade flow wrote the fix and caught a 1000x unit error on the way, the merge request merged itself and shipped through all nine stages. #3, !1
2 A developer copied retired names from an old wiki page. The repair flow fixed their branch. !2
3 A config change brought old names back in production, invisible to CI. Monitor healed production, and the incident flow found the commit and the line. #5, !3
4 A removed CronJob API and a removed CI variable, both provably safe. Code committed the fixes. No agent, zero credits. !4
5 An autoscaler whose fields changed shape. The repair flow moved both targets and kept every value. !5
6 A Vault login with the removed CI_JOB_JWT_V2. The repair flow moved it to id_tokens with an audience. !6
7 The radar found a retry rule GitLab removes next release. The upgrade flow chose the right specific reasons for that job and shipped it, hands-off. #7, !7

Run 1: issue to production

Who it is for

Platform teams: the people who own the clusters, pipelines and dashboards every other team depends on. Today, each upgrade is a migration project somebody has to schedule. With Insurixa it becomes a background process that opens, fixes, ships and audits itself. Adoption is one include line for the CI/CD component and three flows to enable.


How we built it {#how-we-built-it}

The packs

  • insurixa/k8s.py encodes the Kubernetes Deprecated API Migration Guide.[^k8s] It applies apiVersion bumps that keep meaning, sends field changes to review, flags STALE_FIELD leftovers that a find-and-replace leaves behind, and recognises Helm branches that only render the old API on clusters that still serve it, so it does not cry wolf.
  • insurixa/gitlab_ci.py reads GitLab's own removals page, pinned to a commit,[^gitlab] and checks every pipeline against the GitLab version it actually runs on ($CI_SERVER_VERSION). Removed features fail; removals still ahead only warn.
  • insurixa/semconv.py reads the 26-release OpenTelemetry dataset from Attrition.
  • A CI/CD component (templates/insurixa.yml) runs all three as gates. This project includes it exactly as an adopter would, so every pipeline tests the component before anyone else uses it.

The agents, on GitLab Duo Agent Platform

  • Three custom flows with eight agents, versioned as flow registry YAML in .gitlab/duo/flows/: upgrade (assess, upgrade, ship), pipeline repair (diagnose, repair), incident response (investigate, remediate, ship).
  • Triggers: work item created, pipeline failed, mention and assign.
  • Insurixa Steward, a custom chat agent with 8 GitLab tools and the Context7 MCP server for current upstream documentation.
  • Guards that save credits and prevent loops. Each flow's first agent answers SKIP for events that are not its job, and a conditional router ends the flow before any further model call. The repair flow skips pipelines that are not for the merge request's latest commit, stops after two attempts so a person looks, and on NO_REPLACEMENT explains instead of deleting.

The repair flow avoided the label trap

Measuring the agents on real upstream files {#evaluation}

I did not want to grade my own demo. I took six files unchanged from Argo CD, kubernetes/examples and go-redis, put each one in its own merge request, and let the pipeline and the flows handle them with nobody watching. scripts/agent_eval.py grades the final files in code, never with a model: the pod template must be untouched, the selector must select it, CRD names and printer columns must survive, units must be converted, and a file with no upstream replacement must not change at all.

Case What makes it hard Result
!8 Argo CD Deployment apps/v1 needs a selector matching the pod template repaired
!9 New Relic DaemonSet a trap: the object's own labels would select nothing repaired, used the template's labels
!10 Flexvolume DaemonSet privileged host volumes must not change repaired
!11 Argo Workflows, 5 CRDs structural v1 schema per version, printer columns moved repaired
!12 PodSecurityPolicy no replacement API exists refused, explained four options
!13 go-redis pool metrics code made 6 renames, then milliseconds to seconds repaired

==All six passed in one attempt each==, for about 30 GitLab Credits. Six cases show behaviour, not a reliable rate, and I want to be precise about that. With zero failures in \( n = 6 \) trials, the one-sided 95% upper bound[^cp] on the true failure rate is still

$$p_{\text{fail}} \le 1 - 0.05^{1/6} \approx 39\%$$

which is exactly why every agent change in Insurixa is proven by the pipeline and production, not trusted on its own.

Agent evaluation

The repair flow refused correctly

Google Cloud: serves what people use, keeps the record {#google-cloud}

GitLab does the work. Google Cloud serves what people touch and keeps an audit trail. The two are joined by short-lived identity, with ==no stored keys anywhere==:

  • Firebase Hosting serves the documentation, the platform guide and the live playground. The deploy-firebase job publishes them from the same commit the pipeline just tested, uploading only files that changed.
  • Firestore holds the ledger: one record per main pipeline with the stages that passed, the policy verdict, what production emitted and the runner carbon. Security rules make it public to read and writable only by the pipeline's identity.
  • Workload Identity Federation. Each job's GitLab OIDC token is exchanged at Google's Security Token Service for a 10-minute service account token, accepted only for this project on main. A fork or another branch gets nothing.
  • Cloud Run deployment code is in the repository and keyless as well. It waits for a billing account, so the demo service runs on Render's free tier, and I say so everywhere.

Keyless deploy and ledger jobs

Live ledger on Firestore

Why this split is efficient: there is no idle cost (Hosting and Firestore's free tier cost nothing between pipelines), no server between users and data (the playground runs the real Python packs in the browser through Pyodide, and the ledger view reads Firestore directly), Firestore sits in us-east1 next to GitLab's hosted runners, and everything is code (firebase.json, firestore.rules, deploy/gcp/).

Live playground

Built to run against production

  • One resource_group per environment, so two pipelines never change the same environment at once.
  • Verify jobs are interruptible and a newer commit cancels them; deploy jobs never are.
  • Retries for stuck runners and registry errors, a fork guard on the only job that commits, and a monitor that knows when a newer pipeline owns production.
  • Tests an agent cannot weaken: legacy expectations live in frozen fixtures, so the only way to pass is a correct upgrade.

Sustainability, measured with SCI {#sustainability}

I used the Software Carbon Intensity specification[^sci] with the functional unit \( R \) = one upstream change shipped to production:

$$\mathrm{SCI} = \frac{(E \times I) + M}{R}$$

Runner time comes from every job of this project's pipelines through the GitLab API. The coefficients are Cloud Carbon Footprint's published values[^ccf] for Google Cloud us-east1, where GitLab's hosted runners run:

$$C_{\text{runner-hour}} = \underbrace{0.00872\ \mathrm{kWh}}{E} \times \underbrace{576\ \tfrac{\mathrm{g}}{\mathrm{kWh}}}{I} + \underbrace{2.24\ \mathrm{g}}_{M} \approx 7.26\ \mathrm{gCO_2e}$$

Design choice Measured effect
code commits safe fixes, no agent 1.93 instead of 2.30 gCO~2~e per shipped change: 16% less, with zero model calls
SKIP guards end irrelevant flows early a median 45 s instead of 183 s: 75% less runner time per irrelevant event
scheduled pipelines only plan and monitor 64% less runner time than a full pipeline
static hosting and in-browser checks no always-on server for documentation, playground or ledger
the ledger records every pipeline's carbon the trend is visible live, not a one-off claim

Put together:

$$ \text{code-first saving} = \frac{2.30 - 1.93}{2.30} \approx 16\% \qquad \text{guard saving} = \frac{183 - 45}{183} \approx 75\% $$

Model inference is excluded because no public per-request energy data exists. GitLab Credits are the closest public proxy, and all agent work this month, including the evaluation, used 93 of 500. Method and limits: docs/sustainability.md.


Challenges we ran into {#challenges}

  1. Tests that an agent could weaken. An early dry run showed that two tests asserted the demo still contained its old names, so an agent could "pass" by editing the tests. Those expectations moved to frozen fixtures. The only way to pass now is a correct upgrade.
  2. Two pipelines overlapped on main. The newer one deployed first, so the older monitor saw a build it did not expect, rolled back and opened a false incident. The incident flow itself investigated it and reported a false positive, with both commits as evidence. Monitor now checks whether main has moved on before calling anything a regression.
  3. A version bump can still be wrong. In the Prometheus case study, swapping the autoscaler's apiVersion produced an object the API server rejected. The Kubernetes pack now reports STALE_FIELD leftovers, and every manifest goes through kubeconform in strict mode.
  4. Validation that was too strict. Staging first failed on debt that production legitimately carried. It now fails only when staging emits retired names the same build did not emit in CI, so existing debt becomes a planned issue instead of a false alarm.
  5. Triggers fire on everything, and credits are finite. Every issue and every pipeline starts a flow. SKIP guards, stale-pipeline checks and code-first fixes keep model calls for judgement only.
  6. The agent sandbox has no network. Agents cannot install test dependencies, so they say so in their notes and the pipeline does the proving. That is why the telemetry contract and the strict schema check exist.
  7. Security scanning flagged my own code. SAST found unchecked URL opens and a weak random generator. Every URL now goes through one helper that only allows http and https, and identifiers use the secrets module.

Accomplishments that we're proud of

  • A complete loop, not a demo. Production opens the issue, agents fix it, nine stages ship it, monitoring heals production, and governance audits and closes the issue.
  • Seven real runs and a 6 of 6 evaluation on real upstream files, with every issue, merge request and pipeline public.
  • Real findings on public code: three published Prometheus charts that fail to install on supported Kubernetes, and a GitLab pipeline the linter passes while logging in with an empty password.
  • An agent that knows when to stop. On PodSecurityPolicy it changed nothing and laid out four options for a person. I value that refusal as much as any fix.
  • Google Cloud with zero stored keys, and a public ledger that anyone can audit.
  • Measured efficiency: safe fixes at zero model cost, and a carbon figure per pipeline instead of a slogan.

What we learned

  • Agents are most valuable after code has removed every decision that does not need them. The narrower the question, the better the answer.
  • Proof must come from something real: a running service, a real cluster, a strict schema. An agent saying "fixed" is not proof.
  • Knowing when not to act is a feature. Stale pipelines, a second failed attempt and changes with no upstream replacement all deserve a stop, not a guess.
  • Upstream data is underused. Every major project publishes exactly what it removes and what replaces it. Turning that into code is cheap, and it makes AI dramatically safer.

What's next for Insurixa: Upgrades Without Outages {#whats-next}

The vision is a Dependabot for breaking changes. Dependency bots move versions forward. Insurixa makes sure what you wrote against the old version still works afterwards. Every backend system sits on dozens of upstream projects that retire things on a schedule, and each pack is just a pinned table of upstream's own removal data plus a few verdict rules. That format scales to the whole backend stack:

  • More packs: Helm chart APIs, Terraform provider removals, cloud SDK deprecations, Python and Node runtime end-of-life, database driver changes. Each is small, sourced and testable.
  • Community packs: an open format so the maintainers of a project can ship its removal data once, and every team using it benefits automatically.
  • Catalog publishing: the component in the GitLab CI/CD Catalog and the flows in the AI Catalog, so any GitLab project can adopt Insurixa in minutes.
  • Organisation scale: one policy and one ledger across an entire GitLab group, so a platform team can see every upcoming removal before it lands, not after it breaks.
  • Upstream fixes: sending the Prometheus chart corrections back to their maintainers.

The goal is simple: no team should learn about an upstream removal from an outage.


Path B and categories {#path-b}

Requirements checklist:

  • [x] Started from an MIT-licensed open-source project
  • [x] Agentic automation and deployment built on or after 5 October
  • [x] Deployed, with a public URL that stays live until judging ends
  • [x] URL to the original project
  • [x] Optional: live on Google Cloud, with the deployment code in the repository
  • [x] Optional: sustainability quantified with the SCI method

  • Started from my MIT project vignesh2027/attrition, which supplies the OpenTelemetry dataset. Its last commit was 3 October. Everything described above (the packs, the flows, the pipeline, the deployment, Google Cloud, the evaluation) was built on or after 5 October.

  • Entering: Path B Best Hands-off, Most Stages Covered, Most Environmentally Impactful (the SCI section above), and the Google Cloud bonus (insurixa.web.app, with deployment code in deploy/gcp/, firebase.json, firestore.rules and the deploy-firebase and ledger jobs).

  • Live until judging ends: production, the playground, the ledger and the six-hourly pipeline all keep running.

[^bench]: Semconv Drift, a Kaggle benchmark of 17 models graded in code against 26 releases of the OpenTelemetry semantic conventions: https://www.kaggle.com/benchmarks/applesone/semconv-drift [^k8s]: Kubernetes Deprecated API Migration Guide: https://kubernetes.io/docs/reference/using-api/deprecation-guide/ [^gitlab]: GitLab deprecations and removals, pinned at commit 43ecb41e: https://gitlab.com/gitlab-org/gitlab/-/blob/43ecb41ea18fda993c19544a7e59b43657c7994f/doc/update/deprecations.md [^sci]: Green Software Foundation, Software Carbon Intensity specification: https://sci.greensoftware.foundation/ [^ccf]: Cloud Carbon Footprint, Google Cloud coefficients: https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/gcp/src/domain/GcpFootprintEstimationConstants.ts [^cp]: For zero failures in n trials, the exact one-sided Clopper-Pearson bound is 1 - 0.05^(1/n); for n = 6 that is about 0.39.

Built With

Share this project:

Updates

Submission history