OrbitQA Sentinel

AI agent that uses GitLab Orbit's knowledge graph to deliver intelligent test impact analysis, flaky test RCA, and pipeline failure triage — cutting CI time by up to 70%.

Comment

Inspiration

As a QA Automation Lead with 14+ years of experience, I've seen countless CI/CD pipelines grind to a halt because of flaky tests, mysterious pipeline failures, and the dreaded "which tests should I run for this change?" question. Every day, engineering teams waste hours re-running tests blindly, manually triaging failures, and doing detective work that should be automated.

When GitLab introduced Orbit — its AI-powered knowledge graph that understands the relationships between code, pipelines, commits, and contributors — I immediately saw the opportunity: what if we could build an AI agent that uses this contextual graph to make QA intelligent?

What It Does

OrbitQA Sentinel is an AI agent deeply integrated with GitLab Orbit's knowledge graph. It transforms reactive, manual QA processes into a proactive, intelligent system:

  • Intelligent Test Impact Analysis: When a merge request is opened, OrbitQA Sentinel queries the Orbit knowledge graph to understand exactly which components are affected. It then recommends the minimal, high-confidence test suite — reducing unnecessary test execution by up to 70%.

  • Flaky Test Root Cause Analysis (RCA): The agent continuously monitors test history across pipelines. Using pattern recognition over Orbit's historical data, it identifies flaky tests, correlates them to specific code paths or environments, and generates actionable RCA reports automatically posted as MR comments.

  • Pipeline Failure Triage: When a pipeline fails, OrbitQA Sentinel doesn't just report the error — it traces the failure through the knowledge graph, linking it to recent commits, changed files, and historical failure patterns. It surfaces the most likely root cause and suggests a fix path, reducing MTTR dramatically.

  • Natural Language QA Queries: Engineers can ask questions like "Which tests cover the payment service?" or "Show me all flaky tests introduced in the last sprint" — and get instant, graph-backed answers.

How We Built It

OrbitQA Sentinel is built as a GitLab-native AI agent:

  1. GitLab Orbit Integration: The agent connects to GitLab Orbit's knowledge graph APIs to fetch entity relationships — commits, files, test cases, pipeline runs, contributors, and MRs — and uses this rich context for all decisions.

  2. AI Reasoning Layer: Powered by Google Gemini (via Vertex AI), the agent performs multi-hop reasoning over the knowledge graph. It translates natural language queries into structured graph traversals and synthesizes results into human-readable insights.

  3. FastAPI Backend: A lightweight Python FastAPI service orchestrates all agent workflows — receiving GitLab webhook events, querying Orbit, invoking the LLM, and posting results back to GitLab as MR comments or pipeline annotations.

  4. GitLab CI/CD Integration: A custom GitLab CI/CD component makes it trivial to add OrbitQA Sentinel to any pipeline with just a few lines of YAML. The agent runs as a pipeline job, analyzing the MR context and enriching the pipeline with intelligent recommendations.

  5. Agentic Memory: The agent maintains a persistent knowledge base of test history, known flaky patterns, and past RCA outcomes using SQLite, enabling it to improve its recommendations over time.

Challenges We Ran Into

  • Graph Query Complexity: Translating natural language QA queries into efficient Orbit knowledge graph traversals required careful prompt engineering and schema understanding.
  • Flakiness Signal Extraction: Distinguishing genuinely flaky tests from environment-specific failures required building a statistical confidence model over historical pipeline data.
  • Webhook Latency: Ensuring the agent responds to MR events fast enough to be useful (before the developer moves on) required aggressive caching and async processing.

Accomplishments That We're Proud Of

  • Achieved 70% reduction in test execution time on sample CI/CD pipelines by using Orbit-powered impact analysis
  • Built a fully automated flaky test detection system that requires zero manual configuration
  • Created a GitLab CI/CD component that any team can drop into their .gitlab-ci.yml in minutes
  • Demonstrated end-to-end pipeline failure triage with accurate root cause identification in under 30 seconds

What We Learned

  • GitLab Orbit's knowledge graph is a powerful foundation for building context-aware developer tools — the entity relationships it exposes go far beyond what traditional CI metadata provides.
  • Agentic AI works best when grounded in structured, trustworthy data like a knowledge graph, rather than relying purely on LLM reasoning.
  • The biggest wins in QA automation come not from running more tests, but from running the right tests at the right time.

What's Next for OrbitQA Sentinel

  • Multi-project intelligence: Extend Orbit graph traversal across mono-repos and microservice architectures
  • Predictive failure modeling: Train a lightweight ML model on historical pipeline data to predict failures before they happen
  • Slack/Teams integration: Deliver RCA summaries and test impact reports directly to team channels
  • Auto-healing tests: Use LLM code generation to automatically propose fixes for detected flaky tests

Built With

Share this project:

Updates