Incident Commander

Inspiration

As a student getting into software engineering, I quickly realized how intimidating and complex CI/CD pipelines can be. During my research online, I read that developers can spend 20–40 minutes just trying to find the one line of output that explains why a build failed. I realized that the time wasted on debugging scales linearly with the number of pipeline failures. If we represent the total time wasted as $W$, the number of failures as $N$, and the average debug time as $t$, we can express this visually as:

$$ W = \sum_{i=1}^{N} t_i $$

I wanted to build a tool that acts like an experienced senior engineer looking over your shoulder—jumping in the moment a pipeline fails, identifying the exact error, and telling you how to fix it before you even open the terminal.

What it does

Incident Commander is a multi-agent GitLab Duo Flow that serves as an automated debugging assistant. When a CI/CD job fails, it automatically:

1. Triggers: A CI job catches the failure and opens an incident issue.
2. Analyzes: A Log Analyst agent reads the failed job's terminal output, fetches the triggering commit diff, and reads the broken files.
3. Reports & Acts: A Reporter agent posts a structured incident report (including root cause and severity and confidence rating) and creates a new branch (fix/incident-commander-flow) with a TODO pointing exactly to the line that needs fixing.

How we built it

Since I didn't have much prior experience with CI/CD, I spent a lot of time researching online, reading GitLab documentation, and experimenting with API calls. I built the project entirely on the GitLab Duo Agent Platform. It consists of:

• A predefined two-agent YAML Flow.
• A .gitlab-ci.yml when: on_failure trigger job that automatically executes curl requests to the GitLab API. I tailored this script to organize everything smoothly before the AI even starts:
  • It auto-assigns the issue to the commit author by looking up their user ID via $GITLAB_USER_LOGIN and passing it as assignee_ids. The right developer gets the issue in their queue immediately.
  • It embeds the $CI_PIPELINE_URL into the issue description, creating a clickable link so anyone can jump straight to the failed run.
  • It posts the @mention comment (which officially wakes up the GitLab Duo Flow) and explicitly shapes the context—like pipeline_id:${CI_PIPELINE_ID}—making it trivial for the Log Analyst to extract the ID and begin its work.

Challenges we ran into

The biggest hurdle was my lack of experience with advanced CI/CD pipelines. I had to learn how to capture a pipeline failure and organically pass that context to an AI agent.

• The Trigger Mechanism: I initially struggled to get the Flow to trigger automatically. Through trial and error, I found out the bot only wakes up if the @mention is posted in an issue comment, rather than the issue description.
• Prompt Engineering: Getting the AI to reliably associate a stack trace with the actual commit diff and accurately suggest a fix without "hallucinating" took heavy prompt tuning.

Accomplishments that we're proud of

I am incredibly proud of successfully automating complex GitLab API calls to chain a CI failure directly into an AI agent workflow. Reducing the "time to incident report" from 20+ minutes of manual scrolling to ~60 seconds of automated analysis is an accomplishment that could genuinely save development teams hours every week.

What we learned

This hackathon was a massive learning experience. Building this project was a crash course in DevOps and AI orchestration. I learned:

• Workflow Automation: How CI triggers, jobs, and APIs fundamentally operate.
• The internals of GitLab CI/CD (rules, variables, job states).
• How to interact with the GitLab REST API via shell scripts.
• How to design prompts and toolsets for multi-agent systems where one agent's output becomes another agent's input.

What's next for Incident Commander

I'd like to expand its capabilities to analyze cross-project or multi-repository pipeline failures.

Built With

• gitlab-duo
• yml