Minari: Autonomous Flaky-Test Resolution for GitLab

The agent that doesn't just flag a flaky test, it diagnoses, fixes, and verifies it in real GitLab CI, then opens a merge request. Powered by Gemini and the GitLab MCP.

Comment

Inspiration

Every engineering team has flaky tests: tests that pass, fail, then pass again with no code change. They waste hours of re-run time, and worse, they quietly erode trust in CI until people start ignoring red builds. What struck us is that today's tools stop at detection. Datadog CI Test Visibility, BuildPulse, and Trunk will tell you a test is flaky and maybe quarantine it, but they never tell you why, never write the fix, and never prove it's gone. A human engineer still does all of that by hand. We wanted to build the agent that closes that gap: one that doesn't just flag a flaky test, but fixes it, while keeping a human in control.

What it does

Minari is an autonomous agent that runs a full five-stage mission for every flaky test, entirely inside GitLab:

  1. Detect: scores each test with a Probabilistic Flakiness Score (PFS) from its run history.
  2. Diagnose: reads the source and git blame over the GitLab MCP server, grounds itself against similar past diagnoses, and reasons with Gemini to a root cause (race condition, async timing, resource contention, network dependency, data isolation) with a confidence score.
  3. Fix: generates three ranked fix candidates. It will never weaken a test's assertion; that boundary is AST-enforced.
  4. Verify: pushes each candidate to a branch and runs it across five isolated CI runs in real GitLab pipelines, passing only if a variance-reduction safety gate confirms the flakiness is actually gone.
  5. Deliver: opens a real merge request with the full diagnosis and verification report attached. It never merges on its own; when it isn't confident, it files a GitLab issue for a human instead.

Every step streams live to a dashboard so you can watch the agent reason in real time, typically for about one one-hundredth of a cent per fix.

How we built it

  • Reasoning: Gemini 2.5 on Google Cloud, orchestrated as a stateful agent graph with LangGraph. A cost-aware router uses the PFS to pick the right model: Gemini Flash triages simple cases, Gemini Pro reasons over complex failures.
  • The superpower, GitLab MCP: every action flows through GitLab's official Model Context Protocol server (get_file_contents, git_blame, create_branch, push_files, trigger_pipeline, get_pipeline, create_merge_request, and create_issue), so Minari closes the entire DevOps loop without ever leaving GitLab.
  • Safety: tree-sitter parses each candidate to guarantee it's syntactically valid and that the original assertions survive untouched.
  • Backend: Python + FastAPI on Google Cloud Run, with PostgreSQL and Redis. Live agent reasoning is streamed to the browser over Server-Sent Events.
  • Frontend: Next.js + React + Tailwind on Vercel, with a live dashboard, predictive forecasting, and a sustainability view of the compute and engineer-time reclaimed.

Challenges we ran into

  • Streaming a multi-minute mission. Real CI verification takes minutes, and Cloud Run's request timeout kept severing the long-lived reasoning stream. We had to decouple the agent's background task from the SSE connection and design the UI to degrade gracefully.
  • Proving a flaky test is actually fixed. One green run means nothing for a flaky test, so we built a variance-reduction gate that runs the fix five times in isolated containers and compares timing variance before and after.
  • Trusting an AI with a test suite. We refused to let the model touch assertions. Enforcing that with AST analysis, rather than prompting and hoping, was the difference between a toy and something an engineer would actually merge.
  • Taming GitLab MCP responses. Some tools return prose rather than JSON, so we built resilient parsers and a self-healing retry loop.

Accomplishments we're proud of

Minari autonomously opened real, CI-verified, mergeable merge requests across multiple categories of flaky test, each with a transparent diagnosis and a passing five-run verification report, for a fraction of a cent, with a human always in the loop.

What we learned

MCP turns a language model into an agent: the difference between answering a question and finishing a job. We also learned that responsible autonomy is a feature. Being honest about confidence and never merging on its own is what makes the tool trustworthy.

What's next for Minari

Multi-language support beyond Python/pytest, quarantine-and-fix automation on pipeline failure, learning from human review decisions, and a GitHub Actions counterpart.

Built With

Share this project:

Updates