• 

• 

• 

• 

Haka QA Agentic Pipeline — GitLab Transcend Hackathon

Inspiration

A conventional QA pipeline ends with a green or red badge. That is useful, but it is not enough. After the code is written, someone still has to review the changed tests, read CI logs, decide whether a failure is a real regression or external noise, check team policies, and decide if the merge request is safe to release.

That post-code work is slow, manual and easy to skip. Our guiding question was:

What if every merge request had its own QA team, one that explains the evidence and recommends the next decision instead of only reporting pass or fail?

Haka QA Agentic Pipeline hands that work to a team of GitLab Duo agents. The goal is not to replace engineers. It is to give them timely, traceable, evidence-based feedback, and leave only the final merge decision to a human.

What it does

Haka QA Agentic Pipeline is a Supervised agentic QA workflow built on the GitLab Duo Agent Platform. Seven agents and a Phase-2 Supervisor collaborate on every merge request. They comment directly on the MR as GitLab Duo service accounts.

Phase 1: on Merge request → Created (three standalone flows, in parallel)

Agent	Stage	What it does
A1 Consistency Validator	Create + Secure	Validates Gherkin in changed .feature files, scans the diff for hardcoded secrets or a real .env, and lints .gitlab-ci.yml if it changed.
A4 Regression Historian	Plan + Monitor	Reviews MR, commit and issue history to detect regressions, flaky tests and coverage changes. Opens a quality issue for a recurring regression.
A8 Policy Auditor	Govern	Enforces governance policies: MIT license, a required MR label, a test job in CI, open compliance issues. Returns COMPLIANT, WARNINGS or NON_COMPLIANT.

Phase 2: on Pipeline → Passed / Failed (Phase-2 Supervisor)

GitLab CI runs 10 REST API scenarios and stores the detailed report as an artifact. Then the Supervisor delegates to four sub-agents with Supervisor Mode (delegate_task):

Sub-agent	Stage	What it does
A2 Results Analyst	Verify	Interprets the run, classifies each failure as HIGH, MEDIUM or LOW severity, and recommends merge, review or retry.
A3 Dashboard Publisher	Monitor	Downloads the CI artifact and publishes every scenario, with its API request and response, to our Google Cloud dashboard.
A5 Release Readiness	Release	Consolidates A1, A2, A4 and A8 into a GREEN, YELLOW or RED traffic light for the human reviewer.
A6 Artifact Packager	Package	Generates an ISO/IEC 42001 (A.6.2.3) evidence package bound to the pipeline and commit, and publishes it to the dashboard.

The Supervisor makes one adaptive decision. If A2 reports a HIGH-severity failure, there is no trustworthy green result to publish, so it skips A3 and goes straight to a RED verdict. A6 always runs last, because a correct RED block is itself evidence that the control works.

The release gate follows a conservative rule:

$$ \text{verdict} = \begin{cases} \text{RED} & \text{pipeline failed} \lor \text{secret found} \lor \text{HIGH severity} \lor \text{blocking regression} \lor \text{NON_COMPLIANT} \ \text{YELLOW} & \text{only non-blocking observations (flaky tests, performance, WARNINGS)} \ \text{GREEN} & \text{otherwise} \end{cases} $$

When the pipeline fails, GitLab's foundational Fix CI/CD Pipeline flow also runs and proposes a code fix. A failure gets three views at once: A2 explains it, A5 blocks it, and Fix CI/CD Pipeline suggests how to fix it.

MR Created ──► A1 Validator ‖ A4 Historian ‖ A8 Policy Auditor     (Phase 1)
                          │
                CI pipeline runs the API tests
                          │
Pipeline Passed/Failed ──► Phase-2 Supervisor (delegate_task):
     A2 Analyst ─► A3 Publisher (skipped on HIGH) ─► A5 Release Readiness ─► A6 Packager
                          │
               Human approves the merge (Supervised)

Together they cover eight DevSecOps stages: Plan, Create, Secure, Verify, Package, Release, Monitor and Govern.

How we built it

API test layer

We wrote a lightweight REST API suite with the Haka framework, Behave and Gherkin: 10 GET and POST scenarios against DummyJSON. We kept it simple on purpose, so the project stays focused on agent orchestration. The CI job produces a structured JSON report with scenario and step results, method, URL, request and response headers and bodies, durations and errors, and keeps it as an artifact.

GitLab Duo flows

Each agent is a GitLab Duo Custom Flow (YAML in flows/). A1, A4 and A8 have their own MR trigger. The Phase-2 Supervisor has a pipeline trigger and defines A2, A3, A5 and A6 as managed sub-agents inside a single flow. Every agent posts one structured comment, so the reasoning trail lives on the MR next to the code.

Dashboard and cloud

• Cloud Run hosts a FastAPI backend.
• Firestore persists runs and evidence.
• Firebase Hosting serves the dashboard UI: expandable runs, per-scenario status, and a request/response drawer.

A3 runs dashboard/publish_from_artifact.py with run_command. A6 builds the ISO 42001 evidence and publishes it next to the runs.

Challenges we ran into

A flow cannot trigger another flow

GitLab triggers require a human event, so agents cannot chain themselves through mentions. Our first design used separate flows, which ran in parallel and duplicated work. We solved it with a hybrid architecture: independent agents run early on their own trigger, and the post-pipeline agents are orchestrated by a supervisor, where delegation happens inside one flow.

Getting real evidence into the agent

The detailed report lives in the test pipeline. The agent runs in a separate workload with its own pipeline context. Project CI/CD variables are not available there, and the artifact download redirects to object storage, where forwarding the auth header returns a 401. Our publisher script finds the right run-tests job, downloads the artifact without leaking auth on the redirect, and posts the full scenario detail. We refused shortcuts that would lose the request/response detail.

Package registry not reachable from a flow

A6 was meant to upload to the GitLab Generic Package Registry. In a flow runtime, the OAuth token lacks package-write scope and CI_JOB_TOKEN arrives empty. We used the dashboard as the persistent evidence store and tracked a proper registry upload as a follow-up.

Flaky public APIs and platform betas

DummyJSON write endpoints timed out intermittently, so we kept the suite to stable GET and POST calls and taught A2 to tell external instability from regressions. Supervisor Mode is a Beta feature, and one run ended mid-delegation. A pipeline retry recovered it.

Secrets and configuration

An early iteration needed an ingest token inside a flow prompt. Our own Regression Historian flagged it as a risk, so we redesigned the backend so that no secret lives in a prompt or in the repo.

Accomplishments that we're proud of

• Seven agents and a supervisor running on real merge requests, triggered by real GitLab events, with no manual steps between them.
• A genuine adaptive decision: on a forced HIGH-severity failure (MR !12), the Supervisor skipped publishing and returned RED, and Fix CI/CD Pipeline proposed a fix.
• A governance-driven block: in MR !22, every test passed, A8 flagged a missing label, A5 turned the light RED, and A6 still packaged ISO 42001 evidence of that decision.
• The agents review our own work: A1 caught a security issue in a header change, and later pointed out that our Devpost story and video script described different architectures.
• A live Google Cloud dashboard with per-scenario request/response evidence.
• Eight DevSecOps stages covered, with a single human control point.

What we learned

• Specialized agents beat a single mega-agent. Each role has different inputs and a clear output, so the workflow is easier to reason about and to trust.
• Run each agent when its data exists. Diff-only agents should not wait for the pipeline, and result-driven agents cannot run before it. That idea shaped our two-phase design.
• Evidence first. Without real logs, artifacts and history, the right behavior is to state uncertainty, not invent conclusions. Our agents say what they checked and what they could not verify.
• Human-in-the-loop is a feature. Agents do the QA work and recommend. A person makes the merge decision, with all the context already summarized on the MR.
• Observability builds trust. MR comments, Duo sessions, CI artifacts and the dashboard make every step of the reasoning visible.

What's next for Haka QA Agentic Pipeline — GitLab Transcend Hackathon

• Publish A6 evidence to the GitLab Package Registry, with a download view in the dashboard.
• Add trend charts for pass rate, duration, failure categories and flaky tests.
• Use GitLab Orbit more deeply to correlate failures with commits, owners and work items.
• Extend the test layer to web UI, mobile and contract testing.
• Combine QA evidence with security scans and deployment readiness in the release gate.

Built With

• behave
• fastapi
• firebase-hosting
• firestore
• gitlab
• gitlab-ci/cd
• gitlab-duo
• gitlab-duo-agent-platform
• google-cloud-run
• hakaengine
• hakalab
• iso-42001
• mermaid
• playwright
• python
• rest-api