Pipeline Pathologist

Reads the Orbit code graph, not the logs, to diagnose failed pipelines — posting evidence-chained verdicts on the MR: flaky or real, which MR is the culprit, and what's safe to do.

Comment

Inspiration

A pipeline goes red. The on-call maintainer has no context on the three MRs that merged today. The log says KeyError: 'balance' somewhere in a stack trace. A CI agent reads that log and guesses which MR is to blame.

The guess is sometimes right. The problem is that it's a guess, there's no way to check the reasoning, and when the agent is wrong, a developer reverts the wrong MR, reruns the pipeline, and loses another 20 minutes.

The research behind fault localization (Parnin & Orso ISSTA'11; Kochhar et al. ISSTA'16) shows that automated diagnosis outputs are routinely ignored when they cannot be explained. The adoption gap isn't accuracy it's trust. A chain of evidence the maintainer can verify in one click is worth more than a confident verdict that can't be checked.

What it does

Pipeline Pathologist is a GitLab Duo Flow that triggers on pipeline:failed and posts a verifiable evidence chain on the MR:

test_get_account (tests/test_api.py) — failing; KeyError: 'balance'
→ calls handle_get_account (src/api.py:7)
→ calls get_account_summary (src/service.py:8)
→ calls fetch_user (src/db_utils.py:8)  ← modified by this MR

Three behaviors that distinguish it from log-reading agents:

Flaky guard — before any culprit search, the flow checks whether the failing test has an orbit-visible call chain into production code. If calls_into_src = 0, it reads the error text to determine FLAKY vs ENV and stops without blaming anyone. No hallucinated culprit.

Honesty guard — if the call-path files and the MR's changed files don't intersect, the verdict is NO CHANGE-BASED CAUSE FOUND. The flow says so explicitly.

Multi-hop disambiguation — for each file in the intersection, the flow reads the diff and asks: can this type of change produce this type of error? Input coercions can't cause KeyError on return keys. Renamed return keys can. The agent follows the chain to the semantically relevant change, not the nearest one.

How we built it

The core insight: a log-reading agent sees KeyError: 'balance' and searches for where balance is used. The Orbit knowledge graph already knows which functions call which other functions, a recursive CTE over gl_edge with relationship_kind = 'CALLS' produces the full call chain from the failing test in under 5ms on a 15-definition, 64-relationship graph.

All 8 CI-failure agents identified in the hackathon group read job logs and reason over raw text. None traverses a semantic call graph. That's the gap this project fills.

One other graph-aware tool exists in the group: GraphDev (zelong1222) builds a semantic call graph for pre-merge impact analysis. The two are complementary — the post-failure half of graph-native CI intelligence. GraphDev predicts what will break before merge; Pathologist diagnoses what broke after failure.

Stack:

  • GitLab Duo Agent Platform — ambient flow, pipeline:failed trigger
  • Orbit Local (0.74.0) — installed in the agent container via agent-config.yml setup_script; indexed on the project source at startup
  • Tools: get_pipeline_failing_jobs, get_job_logs, run_command (orbit sql), list_merge_request_diffs, create_merge_request_note
  • Demo testbed: Python banking app, 3 modules, 15 functions, 64 call/import relationships

Key engineering finding from the dry-run eval: the gl_edge column is relationship_kind (not kind), and SELECT DISTINCT is required in the recursive CTE functions called multiple times otherwise produce duplicate rows that corrupt suspect ranking. Both are now wired into the canonical flow prompt.

Eval harness

Five failure classes, scored against a ground-truth answer key before any live trigger was available:

Scenario Failure class Orbit signal Verdict Score
S1 Clean culprit 3-hop chain, single intersection CULPRIT FOUND — db_utils.py::fetch_user PASS
S2 Flaky test (~35% random failure) calls_into_src = 0 + "spurious failure" in error text FLAKY — no blame PASS
S3 Multi-suspect (two MRs on the path) 4-hop chain; sign-flip semantics disambiguate CULPRIT FOUND — service.py::transfer PASS
S4 Env failure (missing DATABASE_URL) calls_into_src = 0 + "CI environment issue" in error text ENV ISSUE — no blame PASS
S5 Indirect culprit (bug 2 hops from test) 3-hop chain; hop-1 diff ruled out; hop-2 diff matches KeyError CULPRIT FOUND — service.py::get_account_summary; 1 nearer change ruled out PASS

5 / 5

All five scenarios were subsequently reproduced live on-platform — each verdict posted as a real diagnosis comment on its MR thread.

S2 and S5 are the two proofs a log-reading agent cannot replicate: S2 because the graph has no path (the test calls nothing in src/); S5 because the nearest changed function is semantically innocent and only the graph walk reveals the relevant change two hops deeper.

Challenges we ran into

Schema discovery: gl_edge.relationship_kind is not kind — the column name changed between orbit versions and no public documentation reflected this. Found by running orbit schema against the live graph.

Recursive CTE deduplication: DuckDB returns one row per call site, so a function called twice (e.g., update_balance inside transfer) produces duplicate chain rows without SELECT DISTINCT. This silently corrupted suspect ranking in early test runs.

S2 / S4 same orbit signal: both flaky tests and env failures produce calls_into_src = 0. The disambiguation is entirely in the error text. The flow prompt must check for explicit flaky markers and explicit env markers separately, the graph cannot distinguish these two cases on its own.

What we learned

The recursive CTE over gl_edge is the right tool. A 5ms query over a 64-relationship graph returns a full 3–4-hop call chain with enough signal to distinguish semantically relevant changes from innocent ones. The graph doesn't replace the log; it gives the log a structure to navigate.

What's next for Pipeline Pathologist

  • MCP server for orbit: a Python stdio server exposing orbit_call_chain and orbit_zero_calls_check as proper MCP tools, replacing the run_command shell invocation
  • Companion skill: pipeline-forensics/SKILL.md — a query playbook for manual orbit investigation via Duo Chat
  • Multi-project graphs: once Orbit Remote is available on the group (currently requires an admin feature flag)

Out of scope for v1: auto-revert (suggest only), deep log analysis beyond test-name extraction, and predictive/pre-merge mode (that's GraphDev's half).

Built With

Share this project:

Updates