GhostBreak — Runtime Investigation Lab

Inspiration

Writing code is only the beginning of software development. Some of the most challenging engineering problems arise after the code has been written: reviewing changes, identifying hidden risks, reproducing runtime failures, investigating their causes, and verifying that proposed fixes actually work.

Consider a checkout operation that successfully commits an order but loses its response before reaching the customer. The customer retries the request, and the application unexpectedly creates a second order.

The code may look correct. Conventional tests may pass. Yet a critical business invariant has been violated.

This inspired us to build GhostBreak — Runtime Investigation Lab, an AI-assisted developer platform that investigates what happens when software runs under unexpected conditions.

We built GhostBreak for GitLab's Life After Code challenge, exploring a central idea:

What if GitLab Duo agents could help coordinate the post-code lifecycle, while deterministic execution evidence and human approvals kept the process trustworthy?

Instead of building another AI code generator or recreating GitLab's interface, we wanted to create a complementary investigation environment that connects development automation, runtime risk discovery, fault injection, evidence analysis, and repair verification.

Our guiding principle became:

Reproduce the failure. Understand the evidence. Approve the repair. Verify the outcome.

What it does

GhostBreak is an evidence-driven Runtime Investigation Lab powered by GitLab Duo Agent Platform.

The project brings together three complementary capabilities.

1. GitLab Issue Autopilot — Agentic development automation

We developed a custom GitLab Issue Autopilot agent:

@ai-issue-autopilot-gitlab-ai-hackathon

It supports a human-controlled development workflow initiated through GitLab issue comments.

Its responsibilities include:

• Reading an issue and repository instructions.
• Identifying the requested implementation scope.
• Checking dependencies and existing work.
• Detecting potential duplicate implementation.
• Assessing sensitive changes.
• Requesting human approval when required.
• Working through dedicated feature branches and Draft Merge Requests.
• Reporting implementation results and observed verification evidence.

The agent follows the project's AGENTS.md instructions.

It does not have permission to automatically merge completed work into the integration or release branches.

This provides an auditable development workflow in which AI assists with implementation while the developer retains control.

2. Runtime Risk Explorer — AI-assisted risk discovery

GhostBreak also includes a custom Runtime Risk Explorer flow designed around GitLab Duo Agent Platform.

The flow uses specialized agent responsibilities for identifying runtime risks associated with code changes and communicating proposed investigation scenarios.

Rather than limiting AI assistance to conventional code review, Runtime Risk Explorer focuses on questions such as:

• What could go wrong when this change executes?
• Which unexpected runtime conditions deserve testing?
• What fault could expose a hidden regression?
• Which business invariant should remain true?
• How could the suspected failure be reproduced?

The intended workflow connects GitLab change context with a reproducible GhostBreak experiment.

Scenario proposals are subject to human review before execution.

3. GhostBreak Studio — Runtime investigation and evidence

GhostBreak Studio is the investigation environment.

Its product design is organized around five experiences:

Scenario Lab

Define experiments, fault conditions, preconditions, and deterministic invariants.

Execution Theater

Explore an ordered execution timeline showing application actions, injected faults, and state transitions.

Evidence Inspector

Examine assertion results, expected versus actual values, execution artifacts, and the evidence behind a failure.

Duo Investigation Desk

Connect observed runtime evidence with AI-assisted investigation, root-cause hypotheses, and proposed repairs.

Verification Chamber

Compare original and candidate executions to determine whether an approved repair resolves the observed regression.

Our initial reproducible experiment demonstrates a lost-response checkout failure.

The server commits an order, the response is dropped, and the client retries using the same checkout intent. The defective implementation creates a duplicate order, while the corrected implementation preserves the original order identity.

GhostBreak evaluates deterministic invariants, including:

• Exactly one order must exist for a checkout intent.
• A retry must return the original committed order.

Importantly, GitLab Duo provides reasoning and recommendations; deterministic execution code determines PASS or FAIL.

AI-generated explanations are never treated as proof that a repair worked.

How we built it

We built GhostBreak as a modular application using GitLab as its development and automation foundation.

GitLab and GitLab Duo Agent Platform

Our repository is hosted on GitLab, with an integration branch, dedicated feature branches, Merge Requests, and CI/CD pipeline history.

We created a custom Issue Autopilot agent to support issue-driven implementation.

The agent reads repository instructions, performs preflight checks, identifies sensitive changes, and uses human approval gates.

We also developed the Runtime Risk Explorer custom flow using GitLab Duo Agent Platform flow definitions and specialized agent responsibilities.

Together, these components explore how agents can support multiple post-code lifecycle activities rather than operating as isolated coding assistants.

Deterministic Python execution engine

The GhostBreak engine uses Python to execute controlled runtime experiments.

It supports:

• Defined scenarios and preconditions.
• Deterministic fault injection.
• Positive and negative fixtures.
• Ordered execution events.
• Assertion evaluation.
• Structured evidence generation.
• Reproducible experiment outcomes.

The checkout tutorial demonstrates both defective and corrected behavior.

We are evolving this foundation toward a reusable, versioned execution adapter architecture so developers can define additional supported scenarios without modifying the core engine.

Backend and persistence

We built the backend using FastAPI, SQLAlchemy, Alembic, and SQLite.

Its data model covers scenarios, experiments, runs, evidence bundles, investigations, GitLab references, and verification records.

The REST API exposes resources for retrieving scenarios, launching supported experiments, inspecting execution runs, and accessing their evidence and timelines.

GitLab integration

GhostBreak includes read-only GitLab API integration for retrieving project information, Merge Request context, changed-file metadata, pipeline jobs, and available artifacts.

GitLab remains the system of record for repositories, code review, CI/CD, and merging.

GhostBreak is designed to complement those capabilities rather than duplicate them.

GhostBreak Studio

We developed Studio using React, TypeScript, and Vite.

The interface focuses on making runtime behavior understandable through investigation-oriented views rather than presenting another traditional GitLab dashboard.

Testing and validation

We maintained separate automated test suites for the execution engine, backend, and Studio.

Our latest local stabilization report recorded:

• 194 passing core engine tests.
• 266 passing backend tests.
• 140 passing Studio tests.
• A successful Studio production build.

We also use GitLab development history and CI/CD results to track implementation and regression testing.

Our autonomy model: Path A — Assisted Agent

GhostBreak is submitted under Path A: Start Fresh, using the Assisted Agent autonomy model.

We deliberately chose human-in-the-loop automation.

Our approach is not to give agents unrestricted authority over repository changes, execution environments, repairs, or production releases.

Instead, responsibilities are divided clearly:

Agentic automation	Human responsibility
Analyze issue requirements	Approve implementation scope
Identify dependencies and risks	Resolve uncertain or sensitive decisions
Propose runtime scenarios	Approve experiment execution
Generate investigation hypotheses	Review supporting evidence
Propose implementation or repair	Approve the candidate change
Report test and verification results	Decide whether to merge or release

This design supports traceability and developer control while still allowing agents to perform meaningful post-code work.

We believe trustworthy automation is not measured only by how many decisions an AI system makes independently, but also by how clearly it explains its actions, preserves evidence, and respects approval boundaries.

Challenges we ran into

Coordinating multiple agentic workflows

Building an agent is different from building a reliable agentic workflow.

Our Issue Autopilot needed to understand existing branches, previous work, issue dependencies, repository instructions, and human approval requirements.

During development, GitLab API permission limitations sometimes prevented the agent from verifying issue relationships. Rather than silently assuming that no blockers existed, the workflow stopped and requested explicit human acknowledgement.

This reinforced the importance of transparent failure states in autonomous systems.

Avoiding duplicate or unsafe implementation

An issue-driven agent must not blindly repeat previously completed work.

We introduced checks around existing work, integration branches, and sensitive implementation scope.

We also maintained explicit restrictions against automatic merging and unauthorized changes.

Reproducing runtime failures

Intermittent failures are difficult to investigate because they depend on timing and unexpected event sequences.

We addressed this through deterministic fault points, reproducible fixtures, and explicit invariants.

Separating AI hypotheses from execution truth

A language model may produce a convincing explanation, but that does not establish whether software behaves correctly.

We designed GhostBreak so the execution engine determines test outcomes, while GitLab Duo contributes risk analysis and investigation reasoning.

Connecting development automation with runtime evidence

GitLab issues, Merge Requests, Duo workflows, CI pipelines, backend records, and Studio each represent different parts of the development lifecycle.

Connecting them into a coherent investigation workflow has been one of our central engineering challenges.

Moving beyond a predefined demonstration

Our initial checkout scenario established the deterministic execution foundation.

The next challenge has been generalizing that foundation into a reusable scenario and adapter model without compromising execution isolation or evidence integrity.

Accomplishments that we're proud of

1. Building a custom GitLab Issue Autopilot

We created an issue-driven agentic development workflow with repository-aware instructions, duplicate-work checks, sensitivity assessment, and human approval gates.

2. Developing Runtime Risk Explorer

We designed a specialized GitLab Duo flow focused on identifying runtime risks and proposing investigation scenarios rather than simply generating code.

3. Reproducing a meaningful runtime regression

Our deterministic checkout experiment demonstrates how a lost response followed by a retry can violate an important business invariant.

4. Establishing evidence as the source of truth

GhostBreak evaluates observable execution results instead of allowing AI-generated explanations to determine correctness.

5. Building a full-stack foundation

We implemented a Python execution engine, FastAPI backend, persistence models, GitLab integration components, and React-based Studio.

6. Maintaining automated regression coverage

Our development process includes extensive engine, backend, and frontend tests, alongside GitLab CI/CD history.

7. Keeping developers in control

Our Assisted Agent model requires human oversight for sensitive actions and preserves manual control over code integration.

Most importantly, we developed a product concept that goes beyond another AI coding assistant:

GhostBreak investigates the behavior of code after it has been written.

What we learned

The most important lesson from this hackathon is that generating code is only one part of trustworthy software automation.

The difficult work comes afterward:

• Understanding the actual impact of a change.
• Identifying hidden runtime risks.
• Designing meaningful experiments.
• Reproducing unexpected failures.
• Preserving trustworthy evidence.
• Distinguishing hypotheses from verified results.
• Coordinating human approval.
• Establishing whether a proposed repair genuinely resolves the problem.

We also learned that agentic systems need to communicate uncertainty clearly.

If an API cannot be accessed, a dependency cannot be verified, an execution environment is unsafe, or evidence is missing, the correct response is to report that limitation rather than manufacture a successful outcome.

This principle shaped both our Issue Autopilot and the wider GhostBreak architecture.

Finally, we learned that AI reasoning and deterministic software verification are complementary.

GitLab Duo can help developers decide what deserves investigation. GhostBreak's execution evidence establishes what actually happened.

What's next for GhostBreak — Runtime Investigation Lab

Our next development priorities are:

1. Complete the reusable execution adapter architecture and versioned ghostbreak.yaml scenario configuration.
2. Support additional application scenarios beyond the checkout tutorial.
3. Strengthen the connection between real GitLab Merge Requests and Runtime Risk Explorer.
4. Complete evidence-backed Duo investigation persistence.
5. Enable approved candidate-revision replay with deterministic before/after comparison.
6. Improve Studio's interactive execution timeline and evidence inspection experience.
7. Expand secure execution support for additional application runtimes.
8. Strengthen end-to-end automation while preserving human approval boundaries.

Our longer-term vision is a developer investigation environment that works alongside GitLab throughout the post-code lifecycle.

GitLab manages source code, collaboration, pipelines, and delivery. GhostBreak adds a dedicated layer for understanding runtime behavior and verifying proposed repairs.

GhostBreak — Reproduce what went wrong. Investigate with GitLab Duo. Verify with evidence.

Built With

• alembic
• api
• fastapi
• gitlab
• gitlab-ci/cd
• gitlab-duo
• gitlab-duo-agent-platform
• python
• react
• rest-api
• sqlalchemy
• sqlite
• typescript
• vite
• vitest
• yaml