Impact Engine — Project Story
Tagline: The break that ships green — passes CI, merges clean, takes down prod. A fleet of GitLab agents detects it on the code graph, verifies every claim, and writes the exact fix. Before you merge.
Inspiration
The bugs that cost you aren't the ones a reviewer catches in thirty seconds. They're the ones that pass every gate — green pipeline, clean merge, approved — and still take down production.
Here's the one that inspired this. A developer opens a one-line merge request: tighten how the Money
type handles a zero amount — make is_negative() take an explicit allow_zero flag.
# BEFORE # AFTER (MR !3)
def is_negative(self) -> bool: def is_negative(self, *, allow_zero: bool) -> bool:
return self.cents < 0 ...
Clean diff. Pipeline green. Git reports zero conflicts. A reviewer approves it in thirty seconds. It merges. The next payment 500s. Then every payment.
Why? Because ledger.py — the module that debits and credits every transfer — imports money.py and
calls is_negative() the old way. Nobody touched ledger.py. It wasn't in the diff, so git's
three-way merge never looked at it. CI didn't catch it because the pipeline only compiles (compileall)
— it never runs the call. And the same bare call lives in fraud.py, which no merge request touches
at all. The break lived in the dependency edge between two files, and nothing in the standard
pre-merge toolchain looks there.
This is not an exotic bug. It is the entire class of incident that passes every gate you have and
breaks anyway: a change to one file that breaks a different file nobody edited. And the maddening part:
every fact needed to catch it already lives inside GitLab. The import edge ledger.py → money.py is in
Orbit, the code knowledge graph. The open merge requests are in the API. The CWE finding on the same code
is in the security scanner. The missing test is a query away. They just live in four different places, and
no tool joins them at the one moment that matters — before you click merge.
Impact Engine is the join.
What it does
You ask, in the GitLab Duo Agent Platform: "Assess the change-risk of merge request !3."
A fleet of nine published AI-Catalog agents then each performs one cross-domain join against the live GitLab knowledge graph, an adversarial critic verifies every claim, and you get back one verdict — GO / REVIEW / NO-GO — and the exact patch that makes the merge safe, posted to the MR after a human approves.
| Agent | What it's like | What it actually does |
|---|---|---|
| Semantic Collision Detector ⭐ | the one who sees the invisible | Cross-products open-MR changed files × importers of each changed module. MR !3 changed money.py; ledger.py (edited by !1/!2) and fraud.py (untouched) both call it bare → a confirmed break git can't see |
| Impact Critic ⭐ | the adversarial referee | Re-queries the graph and re-reads the code for every claim. Three verdicts — VERIFIED / REJECTED / UNVERIFIED. Hallucination dies here |
| Remediation Architect ⭐ | the fixer | Turns the confirmed break into the exact, copy-pasteable patch — reads each caller's intent and gives it the right argument. The change-review analog of generated rollback SQL |
| Change Footprint | the migration planner | Who's downstream of a change — every module that must update if you touch this file (the blast radius) |
| MR Collision Detector | the merge cop | Two open MRs editing the same file → a textual conflict git will flag |
| Coverage Gap Scanner | the honest QA | Which risk-path modules have no test importer — so CI never had a chance |
| Finding Intersection | the security joiner | Does this MR's path cross an open SAST finding, and does it fix it or just ship alongside it? |
| Code Owner | the router | Who should look at this — the authors of MRs that last touched the file (honest: MR-authorship, no CODEOWNERS edge) |
| Impact Engine (orchestrator) | the chief of staff | Ties every specialist into one weighted verdict + folds in the patch + opens the human gate |
It synthesizes one verdict — money/auth/security paths count more — and, only after a human approves, posts it as a note on the merge request. For MR !3: 🔴 NO-GO, with the recommended fix attached.
Prove it, don't trust it — then fix it. In the demo we git checkout the MR's branch and press the
same transfer button on a live payments app — every payment returns a genuine HTTP 500 TypeError. That is
exactly what merging would have shipped. Then we apply the Remediation Architect's recommended patch, press
the same button, and it's green again — and the money actually moves (sender debited, receiver
credited). Detect → verify → fix, on a clean, green, conflict-free MR.
Everything lands on the MR. Each agent offers — human-approved — to post its finding straight onto the merge request thread, so the MR accumulates a permanent, per-agent audit trail of what was checked.
How we built it
The architecture — three layers, one knowledge graph
┌──────────────────────────────────────────────────────────────┐
│ Developer's IDE │
│ Antigravity + GitLab Duo Agent Platform │
│ │
│ "Assess the change-risk of merge request !3." │
└───────────────────────────┬──────────────────────────────────┘
│ thin agents → skills auto-load from the default branch
▼
┌──────────────────────────────────────────────────────────────┐
│ Intelligence Layer │
│ GitLab AI Catalog — 9 published agents (Public) │
│ │
│ Semantic Collision · Impact Critic · Remediation Architect │
│ Change Footprint · MR Collision · Coverage Gap │
│ Finding Intersection · Code Owner · Impact Engine (orch.) │
│ │
│ Each agent = a thin system prompt + a skills/*/SKILL.md │
│ that carries the full methodology (auto-loaded at runtime). │
└──────────────┬───────────────────────────┬───────────────────┘
│ query_graph (Orbit MCP) │ GitLab built-in tools
▼ ▼
┌──────────────────────────┐ ┌────────────────────────────────┐
│ GitLab Orbit │ │ GitLab MCP / security tools │
│ Knowledge Graph │ │ │
│ (ground truth) │ │ list_merge_request_diffs │
│ │ │ get_repository_file · grep │
│ ImportedSymbol (imports)│ │ list_vulnerabilities │
│ CALLS (function callers)│ │ get_vulnerability_details │
│ HAS_FILE / HAS_DIFF (MRs)│ │ create_merge_request_note │
│ AUTHORED (ownership) │ │ (the human-gated ACTION) │
└──────────────────────────┘ └────────────────────────────────┘
Layer 1 — the developer asks one question in Duo.
Layer 2 — the agents are thin: a short system prompt that points at a skills/*/SKILL.md, which carries
the methodology and auto-loads from the repo's default branch at runtime. Specialists → critic →
remediation → orchestrator → human gate — the same topology as a code-owned multi-agent system.
Layer 3 — the Orbit knowledge graph is ground truth. Every dependency, collision, coverage, and
ownership claim is a real query_graph traversal — never guessed.
The hero join — the conflict git can't see
Git's merge is textual and per-file. It cannot see that MR-A changing a function in module M breaks MR-B's untouched file that imports and calls it. The Semantic Collision Detector makes that hidden coupling visible with one query — who imports the changed module?
{"query_type":"traversal","node":{"id":"s","entity":"ImportedSymbol",
"columns":["file_path","import_path"],
"filters":{"project_id":{"op":"eq","value":83293701},
"import_path":{"op":"contains","value":"app.core.money"}}},"limit":100}
On MR !3 this returns 11 importer rows straight from the graph — ledger.py, fraud.py,
transfers.py, payments.py, … — read, not guessed. It then reads the actual MR diffs and the call-site
files, finds the bare is_negative() calls, and confirms the break: the second !3 merges, every caller
crashes — and git saw no conflict.
The workflow in action
Developer: "Assess the change-risk of merge request !3."
│
▼
DETECT — Semantic Collision Detector
→ reads MR !3's diff (is_negative gains a required allow_zero)
→ query_graph: who imports Money? → 11 files
→ which open MRs touch them? !1 & !2 edit ledger.py, never fix the call
→ reads ledger.py:16 + fraud.py:15 on main → still bare
→ 🔴 CONFIRMED BREAK · 📋 report
│
▼
VERIFY — Impact Critic (trusts nothing)
→ re-runs every query, re-reads every diff + file, on main AND the !1/!2 branches
→ catches the puzzle: !3's pipeline is GREEN — because compileall only compiles,
it never runs the call. The break is real, just invisible to CI.
→ ✅ every claim verified against the code; whatever it can't prove, it drops
│
▼
FIX — Remediation Architect
→ finds the call sites in the graph, reads each file
→ reads INTENT: ledger rejects a zero transfer (allow_zero=False),
the fraud screener doesn't flag one (allow_zero=True) — not one blind default
→ 📋 fenced-diff patch · recommends the call-site fix
│
▼
ORCHESTRATE — Impact Engine → one weighted verdict 🔴 NO-GO + the patch
│
▼
PROVE → git checkout mr-money-zero → restart → $25 transfer → HTTP 500 (real TypeError)
FIX → apply the patch → restart → $25 → green, money moves (#1 $475, #2 $25)
│
▼
HUMAN GATE → approve → create_merge_request_note → the MR carries the fleet's record
The demo substrate — Aurelia Payments
A small FastAPI fintech backend with a real import graph deep enough to make the joins bite: a Money
value type, a double-entry ledger, a fraud screener, a real CWE-327 md5 finding in config.py,
and four open MRs (!1 idempotency, !2 validation, !3 the zero-policy refactor, !4 security hardening).
A throwaway one-page UI — kept outside the analyzed repo so it never pollutes the graph — shows the
working app (real balances moving on a transfer), then the 500 outage the oracle prevented, then green again.
Challenges we ran into
The graph has more than one way to ask — and only one of them works
To find importers, the agents first improvised neighbors and IMPORTS-edge queries. Both return empty
on this graph — and an empty result looks like "the graph is down," which sent the agent grepping the
repo README and reporting a blast radius "according to documentation." That single behavior — an LLM
reading a writeup instead of the graph — would have gutted the whole "grounded in the code graph" claim.
The fix was a hard directive: the ImportedSymbol entity filtered by import_path is the ONLY importer
query that works; never use neighbors or IMPORTS edges; an empty result is a wrong-query error, not
permission to grep docs. The moment the agent used the right query, the chain healed — 11 rows, no docs.
"0 CALLS edges" was true once, and then it wasn't
We'd documented that Orbit had zero function-level CALLS edges for this project, so we claimed import-level
blast radius only. Mid-build, a live query proved CALLS edges now exist (post_transfer → is_negative,
screen → is_negative). The honest caveat had become a false-modest one that undersold the product
— and risked a contradiction on camera (the detector using a CALLS edge while the critic recited "0 CALLS
edges"). We reframed every assertion: CALLS edges exist but can be incomplete — use them as a lead,
confirm by reading the code.
The critic that was right to doubt, and wrong about a line number
The Impact Critic once rejected a true claim — it said the fraud call was on line 13, not 15 — because
it inferred the call's line from a Definition node's function range instead of reading the file. We
added a rule: read the file, find the exact call line; a Definition range is not the call's line; a
line-number that's slightly off while the break is real is a footnote, never a rejection. The payoff: in
a later run the critic caught the detector's line-number slip and corrected it — then still confirmed
the break. A self-correcting fleet is a stronger story than three agents that happen to agree.
The system prompt is the reliable lever — the skill is not
We learned this the hard way. Skills auto-load from the default branch, so we pushed every guardrail there
— and the agents kept ignoring them. The fix only held when we put the critical steers in the system
prompt (stored with the agent, always in context). Skills are great for deep methodology; anything that
must hold goes in the prompt.
Integrity: teach the method, never the answer
Early skills hard-coded the demo's literal answer (the importer list, is_negative, the line numbers). A
reviewer would rightly read that as a canned demo. We genericized every skill to placeholders, and added a
guardrail forbidding the agents from grounding in any README/comment/prose — derive from the graph and the
code you read, or don't claim it. The demo's 500 is a genuine runtime TypeError; the verdict carries
only graph-verified facts.
Live infrastructure: the Orbit token expires every two hours
The Orbit MCP OAuth token has a 2-hour TTL and the panel caches it in memory, so a fresh disk token isn't enough — the panel must reconnect. We scripted a silent refresh (the refresh-token renews without a browser) and a pre-flight smoke-test, so the graph is provably live before any agent runs on camera.
Accomplishments that we're proud of
It catches the break git structurally cannot. A change to one file that breaks a different file nobody touched — confirmed against the live import graph and the real code, not inferred. We could not find another AI-Catalog agent that does the cross-file semantic-collision join.
The self-correcting fleet. The adversarial Impact Critic re-queries the graph and re-reads every file, caught the detector's own line-number slip, corrected it, and confirmed the substance — and it explains why CI missed the break (compile-only pipeline). That's verification with teeth, not a rubber stamp.
Graph-grounded, on camera. The hero join is a visible ImportedSymbol query returning 11 real rows.
Nothing is mocked; every claim traces to a query_graph result or a file the agent actually read.
Detect → prove → fix, with real money. The demo's 500 is a genuine TypeError; the Remediation
Architect's patch genuinely clears it; and on the working app the balances genuinely move (sender debited,
receiver credited). The whole loop runs on a real FastAPI app, not a hello-world.
A nine-agent fleet + a permanent MR audit trail. Each specialist is independently usable and composes under the orchestrator, and — human-approved — leaves its finding on the MR thread. Open the MR a month later and read the whole story.
What we learned
The system prompt is the reliable lever; skills are the depth. Auto-loaded skills get ignored or cached. Anything that must hold — the query method, the no-docs rule, the output format — belongs in the prompt.
Ground in primary evidence or don't claim it. The single biggest credibility risk was an agent quoting a
writeup instead of the graph. Forcing every claim back to a query_graph result or a file actually read is
what makes "grounded in the code graph" true rather than asserted.
One query is authoritative; the others mislead. ImportedSymbol filtered by import_path is the
importer query; neighbors/IMPORTS return empty and trick the agent into thinking the graph is dead.
Naming the right tool fixed more failures than any guardrail.
Verification needs three verdicts, not two. VERIFIED / REJECTED / UNVERIFIED — a wrong-query empty result is the critic's error, never grounds to reject a true claim. That single distinction stopped the critic from killing real findings.
"Here's the fix" beats "this is risky." The Remediation Architect that reads each caller's intent and hands back the exact patch is the difference between useful and usable — the same lesson generated rollback SQL taught us. A verdict you can act on is worth ten you can only worry about.
What's next for Impact Engine — pre-merge change-risk oracle
Recurse the blast radius beyond one hop — follow the import chain transitively, not just direct callers.
Learn per-path risk weights from merge outcomes — let the orchestrator's weighting (money/auth/security count more) train on which merges actually broke things.
Wire the verdict into a required MR approval rule — a 🔴 NO-GO blocks merge automatically, still human-overridable. The agent recommends; the gate enforces.
Any repo with a knowledge graph. Aurelia is the demo substrate; the joins make no assumption about the domain. Next: validate the import-graph join on a Rails monorepo and a TypeScript service.
Built with
GitLab Duo Agent Platform · GitLab AI Catalog (9 Public agents) · GitLab Orbit (Knowledge Graph) ·
GitLab MCP / query_graph · GitLab security tools · Python · FastAPI (demo substrate). Thin-agent +
skills/*/SKILL.md architecture (skills auto-load from the default branch). MIT licensed. DCO sign-off on
every commit.
Try it out
- Repo:
gitlab-ai-hackathon/transcend/34562572(MIT, LICENSE in About) - AI Catalog (Public): Semantic Collision Detector ⭐ · Impact Critic ⭐ · Remediation Architect ⭐ · Change Footprint · MR Collision Detector · Coverage Gap Scanner · Finding Intersection · Code Owner · Impact Engine — orchestrator
- Demo app: Aurelia Payments (FastAPI) —
aurelia-demo-frontend/serve_demo.py→ http://127.0.0.1:8077/ - Video (≤3:00): link
How it maps to the judging criteria
- Technical Implementation — a 9-agent fleet over a live knowledge graph, a verifiable
query_graphtraversal behind every claim, an adversarial critic that drops unconfirmed ones, and a real, human-gated write-action through the GitLab MCP. - Design & Usability — one question in, one mission-control verdict out; the human gate keeps the operator in control; every agent leaves an auditable trail on the MR.
- Potential Impact — prevents exactly the class of incident that ships green: cross-file breaks, untested risk paths, unfixed findings — on any repo with a knowledge graph — and ships the fix, not just the diagnosis.
- Quality of Idea — the novelty isn't "detect a broken signature" (a type checker does that on one file). It's the cross-domain join at merge time — import graph × open MRs × findings × coverage, assembled in one agent call, then turned into the concrete patch. Orbit-over-MCP is what makes both the join and the call-site-complete fix possible — and no existing pre-merge tool performs it.
Log in or sign up for Devpost to join the conversation.