Test Cartographer

Coverage % treats every line equally. Test Cartographer uses GitLab Orbit's dependency graph to rank untested code by how much depends on it - then writes the tests for what actually matters.

Comment

Inspiration

Every team has a coverage number, and almost no one trusts it. Coverage % treats every line equally - a five-line helper imported by forty modules and a two-hundred-line script nobody calls count exactly the same. So teams either chase 100% (expensive, low value) or ignore the number and the genuinely dangerous gaps, widely-depended-upon functions with no tests, hide in plain sight. GitLab Orbit already knows the dependency structure of the codebase. We realized that structure is exactly the missing signal: fan-in tells you which untested code is worth testing first.

## What it does

Test Cartographer turns the GitLab Orbit code graph into a prioritized testing worklist:

  1. Enumerates every function/method (Definition nodes) in a project.
  2. Measures dependency fan-in by aggregating ImportedSymbol usage across the graph - how many files depend on each symbol.
  3. Detects tests graph-natively - a function is "tested" if a test file imports it (not a filename guess).
  4. Ranks by risk = fan_in × untested (private symbols damped), so the most-depended-upon untested function floats to the top.
  5. Generates the missing unit tests for the top items and opens a merge request.

The output is a structured Test Coverage Map, not open-ended chat. On our demo project it ranks normalize_input (imported by 5 modules, untested) #1 and make_token (imported by 3) #2 - while correctly excluding format_currency (high fan-in but already tested) and deprioritizing handle_request (untested, but nothing imports it, so a test there is low-leverage). That last contrast is the whole point: untested ≠ risky; fan-in is what matters.

After Test Cartographer's procedure produces the missing tests and Orbit re-indexes the project, the same query confirms the loop closed: normalize_input and make_token flip to tested, risk 0, in the live graph.

## How we built it

  • GitLab Duo Agent Platform — a code-defined agent (agents/test-cartographer.yml) published to the AI Catalog, backed by a reusable skill (skills/test-cartographer/SKILL.md) that encodes the procedure and the risk formula.
  • GitLab Orbit — queried through its v2 graph DSL: a traversal to list Definitions, an aggregation over ImportedSymbol by identifier_name for fan-in, and a test-file traversal for coverage detection. Every query is project_id-scoped (Orbit is user-scoped).
  • Model - Anthropic Claude (provided by the Duo platform; we bring no key).
  • Publishing - the official ai-catalog/catalog-sync CI component publishes the agent on a Git tag (0.1.0).
  • Resilience - a dependency-free local AST scanner (tools/local_ast_scan.py) reproduces the same ranking from the working tree, so the workflow still works on an unmerged branch (Orbit indexes the default branch) and is labelled source: local_ast vs source: orbit.
  • Demo corpus - a small layered Python library with intentional, provable test gaps so the ranking is verifiable, not hand-waved.

## Challenges we ran into

  • Orbit indexes the default branch asynchronously, so a freshly pushed branch isn't instantly queryable. We added the local AST fallback and label every finding's source.
  • The graph's IMPORTS/CALLS edges to Definition aren't always resolved, so edge-based fan-in under-counts. We discovered (by querying live data) that aggregating ImportedSymbol.identifier_name is the dependable signal, and built on that.
  • The AI Catalog schema is strict - agents don't take a skills key, tool names must come from a fixed allow-list, and group enablement is handled by the CI component (not a duplicated consumers block). We iterated against the validator until the agent published cleanly.
  • definition_type values are case-sensitive in the graph ("Function"/"Method", not lowercase). Filters silently returned zero rows until we matched the casing.

## Accomplishments that we're proud of

  • The thesis is demonstrably real on live Orbit data, not mocked: the ranking, the tested/untested detection, and the low-rank-entrypoint contrast all come straight from the graph.
  • It closes the loop — it doesn't just report gaps, the procedure produces working tests (committed in tests/test_validators.py and tests/test_auth.py, all 14 tests pass).
  • One-command publish to the AI Catalog via CI; the agent is live and public in the Catalog.
  • A genuine before / after verifiable in the live graph: re-indexing confirms the gaps were closed.

## What we learned

Graph topology is an underused signal in everyday dev work. The same fan-in idea that powers blast-radius analysis is, flipped around, a test prioritizer. Orbit makes that a few queries rather than a research project. We also learned to validate against the live API early - the schema's real behavior (edge resolution, value casing, validator strictness) differs from what the docs suggest, and only running real queries surfaces it.

## What's next for Test Cartographer

  • Multi-language fan-in (Orbit already indexes Go, Ruby, TS, etc.).
  • Symbol-level precision using resolved CALLS edges as they mature.
  • MR-triggered mode: rank only the functions a merge request touches and post the map as an MR note before review.
  • A coverage-debt budget in CI: fail the pipeline if a high-fan-in function lands untested.
  • Auto-MR with generated tests as a turnkey component, so any project can adopt the loop with one tag.

Built With

  • agent
  • gitlab-duo
  • gitlab-orbit
  • pytest
  • python
Share this project:

Updates