Aftershock — Evidence-First Release Safety for GitLab

Safer releases through deterministic policy gates, verifiable evidence, and honest failure diagnosis.

Comment

Inspiration

Building software faster is only part of the challenge. The harder question is how to release it safely.

As AI becomes more involved in software development, I believe release decisions need more than a confident AI response. They need verifiable evidence, clear rules, and a way to handle uncertainty when something goes wrong.

That idea inspired me to build Aftershock: a GitLab-focused release-safety system designed to connect CI/CD automation, security checks, deployment verification, health monitoring, and AI-assisted pipeline diagnosis. I wanted to explore how AI could help investigate failures without being allowed to bypass the rules that keep releases safe.

What it does

Aftershock combines deterministic release policies with GitLab CI/CD automation and an AI-powered investigation flow.

Its capabilities include:

  • Policy-driven release decisions: Evaluates required checks and release conditions before allowing a release to proceed.
  • Security evidence enforcement: Blocks a release decision when required security-scanning evidence is missing.
  • Artifact and deployment verification: Checks that the deployed artifact matches the verified artifact.
  • Health and recovery evaluation: Includes health checks and rollback/recovery verification in its release workflow.
  • Decision records: Maintains a hash-chained ledger of release decisions.
  • AI-assisted pipeline diagnosis: Uses a GitLab-hosted flow to investigate pipeline outcomes and post findings to a merge request.

A key design principle is that AI should assist with investigation, while deterministic rules govern critical release decisions. When the available evidence is insufficient, Aftershock should report uncertainty rather than invent a diagnosis.

How I built it

I built Aftershock around GitLab CI/CD and the GitLab Duo Agent Platform’s flow system. The project combines Python-based policy and evaluation logic, YAML configuration, automated tests, pipeline jobs, and a lightweight operator interface.

I structured the workflow around distinct stages, including validation, security checks, build and deployment, artifact verification, and health monitoring. I also developed a hosted AI flow that can be triggered from a merge request to inspect pipeline outcomes and publish a summary.

I tested the core logic with automated tests and a scripted 24-scenario evaluation. I also tested the hosted flow against real GitLab pipelines. A successful pipeline produced a NO FAILURE FOUND verdict, while a known failed pipeline produced INCONCLUSIVE when the available tool could not identify its failing job.

That distinction helped me evaluate not just whether the system could return an answer, but whether its answer was justified by the evidence it could access.

Challenges I ran into

One of the biggest challenges was connecting the intended diagnosis workflow to the actual capabilities of the hosted GitLab tools.

In a regression test, I supplied a pipeline that GitLab marked as failed, but the available tool returned no failing jobs. Without a job ID, the flow could not retrieve the corresponding logs. Rather than inventing a root cause, Aftershock returned INCONCLUSIVE.

This exposed an important difference between having a sound diagnosis strategy in code and being able to execute that strategy reliably through a live integration.

I also had to think carefully about failure states, missing evidence, allowed-to-fail jobs, and API errors. A system like this must not confuse an absence of evidence with evidence of success. I added regression tests to help prevent misleading healthy verdicts and kept the critical release-safety decisions separate from AI-generated analysis.

Accomplishments that I’m proud of

I’m proud to have taken Aftershock beyond an idea and built a working prototype with a real GitLab integration.

Some milestones I’m particularly proud of are:

  • Building a multi-stage release-safety workflow with security and deployment-verification gates.
  • Passing the project’s GitLab CI pipeline after hardening the diagnosis logic.
  • Creating and triggering a hosted AI flow directly from a merge request.
  • Verifying that the flow could report a successful pipeline accurately.
  • Testing a known failed pipeline and confirming that the flow reported INCONCLUSIVE when it could not obtain sufficient evidence.
  • Adding automated regression tests for cases where incomplete inspection or tool errors could otherwise lead to a false healthy verdict.

As a solo developer, I also had to make the architectural decisions, implement the workflow, investigate integration failures, and validate the results myself. The most valuable outcome wasn’t just getting the happy path to work; it was discovering where the real integration fell short and making that limitation explicit.

What I learned

I learned that reliable automation depends as much on evidence quality as it does on the logic that processes it.

A pipeline can be marked failed while an investigation tool fails to expose the job responsible. An AI agent cannot reliably diagnose information it cannot retrieve, regardless of how carefully its prompt is written.

I also learned to distinguish local test coverage from live integration evidence. Passing unit tests demonstrates that my logic behaves as expected under tested conditions, but it does not guarantee that every hosted tool behaves as expected in practice.

Most importantly, I learned that INCONCLUSIVE can be the correct result. For release safety, an honest limitation is better than a convincing but unsupported explanation.

What’s next for Aftershock

My next priority is improving the reliability and completeness of pipeline investigation. I want Aftershock to identify failed jobs consistently, retrieve the relevant logs, trace failures across downstream and child pipelines where supported, and produce diagnoses grounded in specific evidence.

I also plan to strengthen the end-to-end evaluation with reproducible failure scenarios, including missing security reports, genuine policy violations, deployment-health failures, and recovery verification. Each scenario should demonstrate both the expected decision and the evidence behind it.

Longer term, I want to make Aftershock a useful release-safety layer for teams adopting AI-assisted software development: one that helps developers investigate problems faster without compromising deterministic controls, auditability, or trust.

The goal is not to make AI responsible for every release decision. It is to make release automation more observable, verifiable, and safe.

Built With

Share this project:

Updates

Submission history