Conformity

Inspiration

Standard "AI code reviewers" are probabilistic—they look at code snippets, try to guess if something is wrong, and are easily dismissed as "hallucinating." At the same time, API specs and the code implementing them drift apart constantly: an endpoint disappears from code but stays in the spec, or a route handler is added without being declared in the OpenAPI spec.

We wanted to build something provable, not probabilistic. We realized that by combining the structural knowledge of GitLab Orbit (Knowledge Graph) with a deterministic graph-to-graph diffing engine, we could build a merge-request gate that proves drift rather than guessing at it. This turns the API contract into a real, enforced pipeline gate rather than stale documentation.

What it does

On every merge request, Conformity:

1. Parses the OpenAPI spec (spec/openapi.yaml) to build a contract graph (endpoints keyed by operationId).
2. Queries GitLab Orbit for the implementation graph (the handler function Definitions actually exposed in the code).
3. Performs a deterministic graph-to-graph set diff (contract △ implementation) to identify structural discrepancies.
4. Generates a readable, markdown-formatted report of any violations (e.g., MISSING_IN_CODE or EXTRA_IN_CODE), posts it as a merge request comment, and fails the pipeline.

Because the pipeline fails and GitLab is configured with "pipelines must succeed" (detailed_merge_status = ci_must_pass), the merge request is blocked from being merged until the drift is resolved.

How we built it

We built the project using a modular, decoupled architecture:

• Deterministic Diff Core (scripts/diff_graphs.py): Plain, robust Python with 5/5 passing unit tests. It forms the core engine.
• Contract Builder (scripts/spec_to_graph.py): Parses the OpenAPI specification to extract operationIds and parameters.
• Orbit Code-Graph Builder (scripts/code_graph.py): Executes a targeted traversal query over the live GitLab Orbit API via glab orbit remote query to retrieve handler definitions. It includes a mock mode (ORBIT_MOCK) for offline development and testing.
• Verdict Renderer (scripts/render_block.py): Generates the visual terminal frame (BLOCKED/PASSED), the MR comment, and the audit log.
• The CI Gate (.gitlab-ci.yml): Integrates the components into the GitLab CI pipeline, failing on drift to block merges.
• Proof Bench Dashboard (dashboard/index.html): A static visualization tool deployed via GitLab Pages that lets developers toggle between BLOCKED and VERIFIED states to inspect the graph diff.
• Duo Flow (.gitlab/duo/flows/conformity.yaml): Implements a platform-native Duo Agent flow definition alongside the CI gate.
• Motion Graphic (videos/conformity-motion/): A 7-second verdict animation (MP4 + GIF) for clean visual feedback.

Challenges we ran into

• Orbit Ontology Constraints: We originally planned to query Orbit directly for HTTP routes, endpoints, and parameters. However, we discovered via glab orbit remote schema that Orbit models code structure (Definition, File, calls, imports) rather than HTTP routing concepts.
• The Pivot: To overcome this, we designed a convention: operationId == handler function name. By filtering definitions to functions within routes.py, we joined the spec and the Orbit code graph on this key.
• Handling Decorated Functions: Python routing frameworks use decorators (e.g., @router.get(...)). Orbit classifies decorated functions as DecoratedFunction instead of Function. We updated our targeted Orbit DSL query to fetch both.
• GitLab CI Blockers: In CI, we had to handle authentication carefully (using GitLab PATs with api and read_repository scopes) and navigate GitLab's anti-abuse identity verification gate to ensure jobs run successfully.

Accomplishments that we're proud of

• End-to-End Live Integration: Conformity is fully working and tested against the live GitLab API and GitLab Orbit.
• Zero Hallucination Moat: By separating the verdict (graph set-diff) from the narration (the LLM comment), we built an AI-assisted gate that is 100% reliable. The LLM only formats the message; the math does the blocking.
• Real Merge Gate: Integrating ci_must_pass makes Conformity an actual enforcement mechanism, not just a passive checker.
• Robust Guardrails: We built unit tests ensuring that since Orbit does not expose parameters, the signature diff is skipped during live Orbit runs to avoid false positives.

What we learned

• We learned how to write efficient, targeted graph traversal queries using the GitLab Orbit JSON DSL (v2.9.0) rather than dumping full graphs.
• We discovered the internals of Orbit's ontology and how to map abstract API concepts onto structural programming constructs.
• We gained experience with GitLab Duo Agent Platform flow conventions and writing custom skills (SKILL.md) for agent execution.

What's next for Conformity

• Branch-level Local Indexing: Currently, Orbit indices are point-in-time on the default branch. We want to integrate glab orbit local on the MR branch to catch code-side changes immediately before they land on main.
• Parameter Signature Extractor: Pair Orbit with a lightweight static signature parser to enforce complete parameter-level signature matching.

* Auto-remediation: Use LLMs to auto-generate a patch or MR commits that resolve the drift (e.g., stubbing out a missing handler or updating the OpenAPI spec).

Built With

• gitlab-ci/cd
• gitlab-cli-(glab)
• gitlab-duo-agent-platform
• gitlab-orbit
• python