DEADLOCK — When the pipeline fails, the investigation begins.
Inspiration
Writing code has never been easier. Getting it to work reliably is another story.
With AI becoming increasingly capable of generating entire applications, I started thinking about what happens after the code is written. Someone still has to test it, investigate failures, understand what went wrong, and make sure the fix doesn't introduce another problem.
I found that part much more interesting than generating yet another application.
A failed CI pipeline gives you an error message, some logs, and perhaps a stack trace. But it doesn't necessarily tell you what actually caused the problem, how to reproduce it, or whether a proposed fix genuinely solves it.
I wanted to explore a different approach: what if a failed pipeline could start its own investigation?
That became DEADLOCK.
What it does
DEADLOCK is a Python-based failure investigation and verification system designed to work with GitLab CI/CD and the GitLab Duo Agent Platform.
Its purpose is to connect four things that are often handled separately: detecting a failure, investigating its cause, proposing a repair, and verifying the result.
To demonstrate this, I built a small SQLite booking application with an intentionally vulnerable implementation.
Imagine two people attempting to reserve the last available seat at the same time. Under the wrong conditions, both reservations succeed, even though only one seat exists.
That's the defect DEADLOCK investigates.
The project includes:
- A deterministic concurrency regression test that reproduces the double-booking defect.
- A corrected implementation that passes the same test.
- A Python CLI for running the application, inspecting failures, and generating evidence.
- GitLab CI/CD pipelines that run the tests and preserve their results.
- A custom GitLab Duo flow designed to inspect failing pipelines, investigate source code, propose a targeted repair through a draft merge request, and request verification.
The important part is the evidence. Rather than just claiming that something has been fixed, the system is designed around demonstrating how the same test behaves before and after a correction.
The local verification and hosted GitLab pipelines are working. The custom Duo flow has been created but is awaiting approval, so its live autonomous repair process has not yet been demonstrated.
How we built it
I started with Python, SQLite, and pytest.
The first step was building a functioning booking application. It needed to handle basic reservations, cancellations, and seat availability before I could introduce a meaningful defect.
Then I implemented the vulnerable booking logic and developed a reproducible concurrency test.
Rather than relying on unpredictable timing or adding arbitrary delays, I used controlled execution to make the failure deterministic.
Once that worked, I built the verification and evidence tools around it.
The command-line interface allows the test scenarios to be executed locally, while the evidence system records actual results. GitLab CI/CD runs the same testing logic in a hosted environment.
The repository maintains separate revisions for the vulnerable demonstration and the corrected implementation, making it possible to compare their behavior.
I also developed a custom GitLab Duo Agent Platform configuration, including instructions for retrieving pipeline evidence, investigating the relevant source code, creating a narrowly scoped repair proposal, and verifying the resulting merge request.
A major design consideration was keeping the automation controlled. The repair agent is restricted from changing unrelated files, modifying pipeline configuration, or merging its own proposed changes.
The goal was never to give an AI unrestricted access to a repository. It was to investigate whether useful autonomy could coexist with reproducible testing and human oversight.
The finished project deliberately has no frontend. Most of the interesting engineering happens in the testing, automation, and verification layers.
Challenges we ran into
Making a concurrency bug fail consistently.
This was the first real challenge. Race conditions are notoriously difficult to reproduce because their behavior depends on execution timing.
I didn't want a demonstration that worked only occasionally. Developing a deterministic failure scenario meant I could compare the vulnerable and corrected versions using meaningful, repeatable evidence.
Separating a genuine repair from an apparent repair.
A failing test turning green is encouraging, but it doesn't automatically prove that the underlying problem has been solved.
I had to think carefully about what evidence the system should collect, whether the regression remained unchanged, and how to verify the exact code revision being tested.
Working with GitLab Duo's configuration and permissions.
Building the Python application was only part of the challenge.
Connecting it to GitLab meant dealing with CI configuration, artifacts, custom agent flows, tool permissions, trigger behavior, and approval requirements.
I learned that having a configuration file in a repository is very different from having an operational agent running on the platform.
Preventing the automation from becoming the problem.
An agent that can alter source code also introduces risks.
I added restrictions around which files could be repaired, how often verification could run, and what changes required human approval.
The difficult question wasn't simply whether an agent could make a change. It was how to make that change traceable and reviewable.
Accomplishments that we're proud of
The most satisfying achievement was getting the complete failure demonstration working in real GitLab pipelines.
The vulnerable version produced a failed hosted pipeline, with the regression trace confirming that the last available seat had been sold twice.
The corrected version produced a successful hosted pipeline.
That was an important moment because the result wasn't just something displayed by a local script. GitLab independently ran the code and reported the outcomes.
Other milestones include:
- 55 passing local tests in the latest verification run.
- 100% booking-service test coverage in the earlier coverage audit.
- A deterministic concurrency regression instead of an unreliable timing-dependent demonstration.
- Working GitLab CI/CD pipelines for both vulnerable and corrected versions.
- Automated generation of authentic failure and verification evidence.
- A custom GitLab Duo flow registered for approval.
- Repair safeguards designed to preserve human review and prevent uncontrolled changes.
I'm particularly pleased that DEADLOCK remained relatively small. I could have built a dashboard, added unnecessary services, and made the project look more complicated. Instead, I focused on getting the core engineering right.
What we learned
The biggest lesson was that generating a solution and proving a solution are two very different things.
AI can suggest a plausible correction, but confidence should come from independently executing tests and examining the results.
I also learned a lot about concurrency, SQLite transaction behavior, automated testing, GitLab CI/CD, and the practical challenges of integrating agentic systems into a development workflow.
One particularly useful lesson was the importance of designing the failure scenario before designing the automation.
Once I had a reliable way to reproduce the defect, everything else became easier to reason about. The test provided a clear target for the investigation and a concrete way to evaluate the proposed repair.
I also came away with a greater appreciation for controlled autonomy. Letting an agent do more isn't automatically better. In some situations, the more valuable feature is knowing exactly what the agent is allowed to do, what it actually did, and when a human needs to intervene.
What's next for DEADLOCK
The immediate next step is to get the custom GitLab Duo flow approved and execute a complete live investigation.
I want to demonstrate the entire process inside GitLab: a failed pipeline, automated evidence retrieval, root-cause investigation, a targeted repair proposal, and successful verification on the exact repaired revision.
Beyond that, I'd like to extend DEADLOCK to investigate different classes of failures, including database transaction issues, dependency regressions, and intermittent test failures.
Eventually, it could support multiple projects rather than a single demonstration application.
I also want to experiment with stronger verification methods and better ways to explain why an agent believes a particular fix is correct.
But the fundamental idea would remain the same.
Software shouldn't be considered fixed just because someone, or something, says it is. It should be able to prove it.
Log in or sign up for Devpost to join the conversation.