QualityGraph

QualityGraph turns API changes into evidence-backed impact maps—revealing affected rules, tests, and missing coverage without treating AI output as truth.

Comment

Most AI testing tools generate more tests. QualityGraph first proves what changed, why it matters, which tests are affected, and what coverage is still missing—all with attributable evidence and a human-governed truth boundary.

Inspiration

API quality work rarely begins with one clean specification. Teams receive contracts, business guidance, rules, test inventories, automation metadata, and historical failures. The hard part is not storing those files—it is knowing which conclusions they support, where they conflict, and whether existing tests actually cover a proposed change.

We built QualityGraph to make that reasoning visible and attributable without allowing generated output to become automatic truth.

What it does

QualityGraph ingests a deterministic synthetic commerce bundle and produces two governed views:

  • Business Understanding Readiness identifies understood concepts, unresolved terms, source conflicts, missing ownership, missing rules, and a reproducible confidence result.
  • API Change Impact compares refund contract v1 with v2 and traces a required reason_code field to Order, Payment, Refund, two business rules, three tests, one historical failure, and two uncovered validation scenarios.

The judge-facing page presents the complete story in three sections—Understand, Change, Coverage—with evidence locators and a clear human-review boundary. It works without a model key and can also exercise the real module-scoped read APIs.

How we built it

  • Python 3.12, FastAPI, Pydantic, SQLAlchemy, Alembic, and asyncpg
  • React 19, TypeScript, Vite, and generated OpenAPI types
  • PostgreSQL as the system of record
  • Neo4j as a rebuildable projection
  • Temporal with separate workflow and activity workers
  • Playwright, Vitest, Pytest, Ruff, Mypy, and Import Linter
  • Docker Compose for the resettable local judge stack

The fixture contains nine fully synthetic source artifacts. Deterministic capabilities run behind governed query boundaries, while action, promotion, and external integration paths remain disabled for the demo.

What we built during the hackathon

The immutable pre-build-week-baseline tag identifies the pre-existing generic platform. Build Week work is isolated in later dated commits and includes the synthetic commerce fixture, Business Understanding Readiness, refund v1-to-v2 impact traversal, focused responsive judge page, one-command resettable Compose runtime, browser acceptance path, and submission documentation.

How we used Codex and GPT-5.6

Codex/GPT-5.6 accelerated architecture exploration, cross-layer implementation, test generation and repair, accessibility-aware UI work, Docker/Temporal orchestration, and repeated quality/privacy gates. Human decisions defined the product story, synthetic fixture size, competition scope, governance invariants, review boundary, and which external actions must remain disabled.

The repository includes a dated CODEX_COLLABORATION.md with verifiable file sets, decisions, failures, corrections, and test results.

Challenges

  • Preserving the distinction between candidate evidence, review approval, and effective Truth.
  • Making impact traversal explainable rather than presenting an opaque score.
  • Building a three-minute experience without hiding uncertainty or source conflicts.
  • Keeping the demonstration useful without model access or private infrastructure.
  • Separating pre-existing platform work from competition-period features with an immutable tag and dated commits.

Accomplishments

  • A 74.72% readiness assessment backed by nine attributable synthetic sources.
  • A reproducible change graph with 12 evidence-backed edges.
  • Exact visibility into three affected tests and two genuine coverage gaps.
  • One browser test that crosses the real FastAPI boundary and validates the complete judge story.
  • A resettable local architecture that preserves PostgreSQL authority, rebuildable Neo4j, and I/O-free Temporal workflow code.

What we learned

The most valuable output is often not another generated test. It is a precise, evidence-backed statement that a business term is undefined, two sources disagree, ownership is absent, or a new validation behavior has no test. Governance makes model-assisted reasoning more useful because it makes uncertainty and authority explicit.

What's next

  • Validate the complete Compose start/reset path from a clean Docker-capable host.
  • Add a judge-accessible sandbox after Docker-capable launch/reset validation.
  • Extend the synthetic demonstration to additional change families while preserving attribution.
  • Add human review workflows for resolving conflicts and approving—but never automatically promoting—candidate knowledge.

Testing instructions

From the repository root:

./scripts/demo.ps1 start    # Windows
./scripts/demo.sh start     # macOS/Linux

Open http://localhost:3000. Detailed testing and reset steps are in docs/INSTALLATION_AND_TESTING.md.

Built With

Share this project:

Updates