UpToCode

Evidence-backed architecture review for AI agents—from deterministic findings to an approved Codex plan.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for UpToCode

Inspiration

Agent code can be syntactically valid and still ship with an unbounded loop, no run budget, an ungated destructive tool, or no eval coverage. Those are architecture defects, not lint errors. UpToCode was built to catch them early, explain the evidence, and keep a human approval step between analysis and source changes.

What it does

UpToCode is an evidence-backed architecture-quality scanner for Python and TypeScript AI agent applications. It recognizes OpenAI Agents SDK, Anthropic, CrewAI, PydanticAI, LlamaIndex, LangGraph, MCP servers, and conservative custom agent loops.

It can:

  • detect missing execution bounds, budgets, approval gates, validation, resilience, evals, observability, and other agent-specific controls;
  • emit terminal, JSON, standalone HTML, GitHub annotations, job summaries, and SARIF 2.1.0;
  • show analyzed-file coverage, source evidence, stable fingerprints, and primary-source citations instead of inventing a quality score;
  • redact recognized secrets and narrow PII before reports or optional model judgment;
  • bind human approvals and rejections to an exact report fingerprint;
  • generate an approved-only FIXPLAN.md for Codex without changing source;
  • expose ten typed local MCP tools for source, file, repository, diff, loop, schema, catalog, review, and planning workflows;
  • provide a smaller credential-protected hosted MCP surface for submitted-content testing;
  • integrate with pre-commit, GitHub Actions, SARIF code scanning, and a VS Code LSP extension.

Static scanning is local, offline, and non-mutating. The optional GPT-5.6 judgment tier is double opt-in (--judgment --send-code), sends only bounded redacted evidence, uses OpenAI Responses Structured Outputs with store=false, and cannot erase deterministic findings.

How we built it

The core is Python 3.11+ with AST analysis, tree-sitter TypeScript parsing, Pydantic contracts, Typer, pygls, the official MCP SDK, and OpenAI Responses Structured Outputs. The repository also contains a TypeScript VS Code extension, a Next.js product site, a composite GitHub Action, and a credential-protected Cloud Run MCP deployment.

Codex collaborated throughout the project: translating the specification into vertical slices; implementing and reviewing the scanner, CLI, reporters, MCP surfaces, site, and tests; fixing clean-clone defects; and running Windows and Linux release gates. The work order, decisions, progress log, and validation evidence remain in the repository as an auditable record. UpToCode also closes the loop by generating a report-bound, approved-only plan designed for a safe Codex hand-off.

GPT-5.6 powers the optional contextual judgment tier. Deterministic analysis remains the default and the offline judge path requires no API key or paid model call.

Challenges

The hardest problem was being useful without overclaiming. Dynamic Python and TypeScript constructs can hide intent, so unsupported cases become explicit coverage gaps or analysis warnings rather than false “clean” verdicts. We also had to keep reports share-safe, preserve findings when model judgment fails, bind review decisions to the exact report, and make local and hosted MCP boundaries testable without exposing repository paths or payloads.

Accomplishments

  • One deterministic scan engine powers CLI, reports, CI, LSP, and MCP surfaces.
  • Review manifests are fingerprint-bound and FIXPLAN generation includes only approved findings.
  • The hosted service exposes submitted-content tools only, with bearer authentication, rate limiting, judgment disabled by default, and payload-free logs.
  • The project audits itself at the release gate with zero production findings, suppressions, or unexplained analysis warnings.
  • The full offline suite and acceptance runner exercise real CLI and MCP lifecycles without requiring a live model call.
  • The final local verification on July 21 passed all 264 tests, the real stdio MCP acceptance path, and the architecture compliance self-scan.

What we learned

Architecture-quality tooling needs three layers: conservative machine evidence, explicit human judgment, and a bounded implementation hand-off. Collapsing those layers into an auto-fix makes findings harder to trust. We also learned that coverage and provenance are product features: a scanner must say what it analyzed, what it could not prove, and which exact evidence supports every verdict.

What's next

Next steps are broader framework coverage, richer data-flow evidence, more editor actions that preserve explicit approval, and production hardening of the hosted judge surface. Source-changing automation will remain opt-in and isolated from the invoking checkout.

Built With

Share this project:

Updates