Keystone — Cross-Project API Evolution Engine

Change one function signature, and Keystone shows you every project it breaks across your whole GitLab group, then orders the migration, generates the shim, and posts it all on the merge request.

Comment

Inspiration

A teammate changed one function signature in a shared library:

- def charge(amount):
+ def charge(amount, currency):

That single line broke code in three projects he had never opened. We only found out when an unrelated pipeline went red two repos away. The fix took an afternoon. Finding out what broke took two days of grepping repo by repo and asking people in chat.

The thing that stuck with us: every code-intelligence tool we tried runs on a single repository. They index one repo, answer questions about one repo. None of them can answer the only question that actually mattered here, which is "who, across the entire group, depends on the thing I just changed?" That is not a missing feature. It is a structural limit. A tool that only sees one repo cannot reason about the other forty.

GitLab Orbit Remote holds the whole group's code graph in one queryable place. That is the missing primitive. So we built the tool that the primitive makes possible.

What it does

Keystone watches library merge requests. When one changes a public API, it runs a seven-stage pipeline and posts the entire result as one comment on the merge request. An action on the SDLC surface, not a chat window.

  1. DETECT. Diffs the public contract, not the textual signature line. It parses both sides into a semantic model (functions, classes, methods, dataclasses, default values, raised exceptions) and classifies the change as breaking | risky | safe. It also catches behavioral drift: same signature, different body, which a signature diff misses entirely.

  2. RESOLVE. This is the moat. It reconstructs the cross-project consumer graph that Orbit does not bridge natively. It walks ImportedSymbol edges, scores each candidate against the defining project, and rejects name-collision decoys. If two projects both import a symbol called charge, only one of them is the real consumer. Keystone tells them apart.

  3. ORDER. Builds the consumer dependency graph and produces a safe migration order with Tarjan strongly-connected-components plus a topological sort. Dependency cycles collapse into a single group instead of crashing the sort.

  4. RISK. Applies one non-negotiable safety law: never recommend an auto-merge on a call path with no test coverage, and never on a behavioral change. The policy is data, not vibes.

  5. EVOLVE. Generates an expand/contract shim that makes the breaking change non-breaking, plus best-effort patches for each consumer call site.

  6. VERIFY. A bounded CI self-heal loop: trigger, diagnose, patch, repush, up to three attempts, then escalate to a human. The loop is allowed to give up. That is a feature.

  7. REPORT. Posts one campaign comment carrying the whole thesis: the contract change, the live cross-project consumer closure, the migration order, the safety verdict per consumer, the shim, and the decoys it excluded and why.

It runs live, on a real merge request

On payments-core!1, charge(amount) becomes charge(amount, currency). keystone run --mr 1 --post finds billing (direct consumer) and invoicing (two project hops away, spanning three repos), correctly excludes notifications (which imports a different function also named charge), orders the migration billing -> invoicing, escalates both consumers because no test covers the affected call path, and posts it all as one comment.

How we built it

The whole engine is plain Python 3.12 with zero runtime dependencies. The only third-party packages are pytest, pytest-cov, and ruff, and they never ship. That was a deliberate constraint: a tool that runs inside a CI job and a Duo flow should not drag a dependency tree behind it, and it forces the design to stay honest about what it actually needs.

The architecture is one spine with pluggable ends. Every analysis mode reduces to the same primitive: seed -> traverse -> check -> order -> act. The contract checker is one plugin behind a ContractExtractor registry. We shipped a second extractor for OpenAPI/REST to prove the point, so the same resolver, ordering, and report machinery handles a Python signature change and a REST schema change without knowing the difference. The cross-project resolver is the part we refused to touch once it was correct.

Modules, each built test-first in isolation:

  • orbit/ - typed client for the Orbit Remote knowledge graph. We reverse-engineered the real multi-node CALLS traversal from the live DSL.
  • resolve/ - the publisher index and scored resolver (the moat), plus transitive reverse-BFS for the full consumer closure.
  • detect/ - AST-based semantic contract model, the breaking | risky | safe classifier, and the behavioral-change signal.
  • order/ - iterative Tarjan SCC plus topological sort.
  • risk/ - test-coverage signal and the safety law.
  • evolve/ - executably-correct expand/contract shim generation and consumer call-site patches.
  • verify/ - the bounded CI self-heal loop.
  • report/ - campaign markdown, including an inline Mermaid blast-radius DAG with severity colors.
  • eval/ - a precision/recall harness scored against a ground-truth fixture.
  • gitlab/ - merge-request and CI operations with idempotent comment posting.

It ships to the GitLab Duo Agent Platform as a custom flow. A read-only research_agent queries Orbit with glab orbit and feeds the writer agent, which keeps the implementer's context lean instead of dumping the whole graph into one prompt.

System architecture

flowchart TD
    MR["Library merge request<br/>(public API changed)"] --> DETECT

    subgraph ENGINE["Keystone engine - zero runtime deps"]
        direction TB
        DETECT["DETECT<br/>AST contract diff<br/>breaking / risky / safe<br/>+ behavioral signal"]
        RESOLVE["RESOLVE (the moat)<br/>cross-project consumer closure<br/>reject name-collision decoys"]
        ORDER["ORDER<br/>Tarjan SCC + topological sort"]
        RISK["RISK<br/>safety law:<br/>no auto-merge without test coverage"]
        EVOLVE["EVOLVE<br/>expand/contract shim<br/>+ consumer call-site patches"]
        VERIFY["VERIFY<br/>bounded CI self-heal<br/>trigger to diagnose to patch to repush (max 3) to escalate"]
        REPORT["REPORT<br/>one campaign comment<br/>+ Mermaid blast-radius DAG"]

        DETECT --> RESOLVE --> ORDER --> RISK --> EVOLVE --> VERIFY --> REPORT
    end

    ORBIT[("GitLab Orbit Remote<br/>knowledge graph<br/>(whole group)")] -. "Definition / ImportedSymbol<br/>CALLS traversal" .-> RESOLVE
    REGISTRY["ContractExtractor registry<br/>Python AST and OpenAPI/REST"] -. plugin .-> DETECT
    REPORT --> POST["Comment posted on the MR<br/>(action, not chat)"]

    subgraph FLOW["GitLab Duo Agent Platform"]
        RESEARCH["research_agent (read-only)<br/>glab orbit queries"] --> WRITER["keystone_agent<br/>runs the engine"]
    end
    WRITER --> DETECT
    ORBIT -. queried by .-> RESEARCH

How it is measured

A demo that says "trust me" is not engineering. The resolver is the load-bearing claim, so we score it against a ground-truth fixture that deliberately contains a name-collision decoy. That means the number measures discrimination, not matching. Anything can match an obvious consumer. The hard part is rejecting the project that imports a same-named symbol from somewhere else.

$ keystone eval --group nexthire-ai-group/keystone-fixture --project 83598460 \
    --fqn payments_core.core.charge --groundtruth tests/fixtures/keystone-fixture/_meta.json

Keystone resolver eval (collision-decoy fixture)
  precision : 1.00
  recall    : 1.00
  f1        : 1.00
  confusion : TP=2  FP=0  FN=0  TN=2

Recall 1.0 means no true consumer was missed. Precision 1.0 with the decoy present (TN on notifications) means it did not fire on the lookalike. Run live against Orbit Remote.

Challenges we ran into

  • Name-collision decoys. The first naive resolver matched on symbol name and happily flagged notifications as a consumer of charge. It was importing a completely different charge. The fix was to resolve every import back to its defining project and score on that, which is the entire reason the moat exists.

  • Orbit returns a truncated graph, and pretending otherwise is a lie. The traversal has a result limit. We raised it from 100 to 500, but more importantly we track saturation and emit a per-change recall caveat when a result set hits the ceiling. The tool knows when it might be incomplete and says so, scoped to that one change so it never bleeds across changes.

  • The decidability wall. getattr, importlib, __getattr__, globals().get: dynamic dispatch is not statically resolvable, full stop. Instead of guessing, resolve/dynamic.py detects these patterns and flags them as an explicit decidability note in the report. Knowing exactly where the wall is, and being honest about it, beats a confident wrong answer.

  • Letting an LLM near a merge. We added an LLM advisory layer for severity hints, but it never mutates the deterministic ChangeKind, and in the self-heal loop the CI gates the LLM, not the other way around. A hallucinated fix fails CI and gets escalated. It is structurally incapable of merging a fix that does not pass. Adversarial review of this path caught a real bug where a NUL-byte fix payload could slip through, which we now reject.

  • Surviving malformed input. A code review found that ast.parse crashed the whole run on a single malformed file in the group. One bad file should not take down an org-wide analysis. It is now contained and reported, not fatal.

  • Behavioral change with an identical signature. A function can keep its exact signature and change what it does. A signature diff sees nothing. Our contract model carries a behavioral signal so this lands in risky instead of silently passing as safe.

What we learned

The interesting leverage was not in any single algorithm. It was realizing that leak tracing, contract drift, refactor impact, and self-heal are the same shape: seed -> traverse -> check -> act. Once we saw that, the resolver and ordering became shared infrastructure and the "what are we checking" question became a plugin. That is why adding OpenAPI support was a new extractor and not a new product.

We also learned that the honest answer is the impressive one. The features we are proudest of are the saturation caveat and the dynamic-dispatch note, because they draw a hard line around what the tool can and cannot decide. Cross-project analysis is only useful if you can trust it, and you can only trust it if it tells you where it stops.

What's next

  • More contract extractors behind the same registry: GraphQL schemas, protobuf, database migrations.
  • Auto-opening the consumer migration MRs in dependency order instead of describing them.
  • Feeding the campaign report back into the knowledge graph so blast radius becomes a first-class, queryable property of every public symbol.

Field: Built with

Comma-separated, lowercase (Devpost renders each as a tag):

python, gitlab-orbit, orbit-remote, knowledge-graph, gitlab-duo-agent-platform, glab-cli, ast, tarjan-scc, topological-sort, gitlab-ci, gitlab-merge-requests, mermaid, pytest, ruff, openapi, ai-catalog

The load-bearing tags are gitlab-orbit, orbit-remote, knowledge-graph, and gitlab-duo-agent-platform. The rest show range.

Field: Try it out links

GitHub repo (MIT):      https://github.com/theCodeForgerHQ/gitlab-showcase-track
Live demo MR:           https://gitlab.com/nexthire-ai-group/keystone-fixture/payments-core/-/merge_requests/1
AI Catalog flow:        <paste public AI Catalog flow URL after publishing>
Demo video (<=3 min):   <paste public, no-copyrighted-audio video URL>

Image gallery shot list (3:2, up to 15, 5 MB each)

Lead with proof, then architecture, then receipts:

  1. The campaign comment on the live MR. Full screenshot of payments-core!1 showing the posted Keystone report. Single most convincing image: it runs on a real GitLab surface.
  2. The cross-project consumer closure table from that comment: billing direct, invoicing hop 2, notifications excluded as a decoy. The decoy line is the moat made visible.
  3. The Mermaid blast-radius DAG rendered from the report, severity colors and all.
  4. The system architecture diagram (render the Mermaid block above to PNG).
  5. The eval terminal output: precision/recall/f1 = 1.00, TP=2 FP=0 FN=0 TN=2. Caption: "scored against a fixture that contains a decoy, so this measures discrimination."
  6. The diff that starts it all: charge(amount) becoming charge(amount, currency), three lines, clean.
  7. The flow in the GitLab Duo AI Catalog once published, showing Public visibility.
  8. A tree of src/keystone/ captioned with zero runtime dependencies and 352 unit tests.

Built With

  • gitlab-api
  • gitlab-ci
  • gitlab-duo-agent-platform
  • gitlab-merge-requests
  • gitlab-orbit
  • glab-cli
  • knowledge-graph
  • pytest
  • python
  • ruff
Share this project:

Updates