ArchitectAgent — Self-Healing Repository Auditor

Inspiration

A pull request can look correct while quietly breaking something it never touched.

Imagine a developer changes a pricing function to return cents instead of dollars. The function itself works as intended, and the pull request may look harmless. But an untouched checkout module still interprets the returned value as dollars, creating a regression somewhere else in the repository.

This is the kind of problem that inspired ArchitectAgent.

I wanted to explore how AI could help developers reason beyond the files directly changed in a pull request—to trace potential cross-file impact, explain the evidence behind a finding, propose a targeted repair, and determine whether that repair passes relevant tests.

The goal was not to build another chatbot that reviews code. It was to build an engineering tool that connects AI-assisted reasoning with repository evidence and controlled validation, while keeping the developer in control.

What It Does

ArchitectAgent is an evidence-grounded, AI-assisted GitHub pull-request auditor that helps identify potential cross-file regressions.

Its workflow brings together four capabilities:

• Trace impact: Analyze repository relationships and identify code that may be affected even when it was not directly modified.
• Explain findings: Produce reports that connect potential regressions to repository paths, import relationships, code excerpts, and relevant test assertions.
• Propose repairs: Generate targeted regression-test and patch proposals for developer review.
• Validate changes: Check patch scope and application, and use an appropriately configured isolated runner to execute supported tests and record their actual outcomes.

ArchitectAgent combines deterministic analysis with an optional AI pipeline rather than treating a model's explanation as proof. Its reports distinguish repository-backed evidence from AI inference, and proposed repairs remain subject to validation.

ArchitectAgent does not automatically push commits or merge pull requests. Its purpose is to help developers make better-informed decisions, not to remove them from the review process.

How I Built It

I built ArchitectAgent in Python using FastAPI, with a backend designed around reliable, asynchronous processing.

A verified GitHub webhook receives supported pull-request events. PostgreSQL stores durable scan state, while an outbox and Redis-backed RQ workers support asynchronous processing and recovery. GitHub App authentication is used to retrieve relevant pull-request and repository information.

The analysis engine combines Python AST inspection with Tree-sitter-based JavaScript and TypeScript import analysis. This deterministic layer provides repository structure and evidence that can ground subsequent reasoning.

For AI-assisted analysis and generation, ArchitectAgent integrates Nebius Token Factory with NVIDIA Nemotron 3 Ultra and Nemotron 3 Super. The configured two-stage pipeline uses the analysis model to reason about potential impact and the generation model to propose targeted regression tests and patches.

Model outputs are validated against strict schemas and repository evidence before being incorporated into the workflow. Patch validation checks the proposed changes against scope constraints, while stale-pull-request protections help prevent results for an outdated revision from being treated as current.

The results are published as a single updated pull-request comment, containing findings, impact information, proposed tests and patches, validation statuses, and Mermaid-based relationship diagrams.

Secure Sandbox Integration

One of the central engineering challenges was deciding how generated tests and patches should be evaluated without executing untrusted repository code inside the main application.

ArchitectAgent includes a separate sandbox controller and an optional Docker-based runner for controlled execution of supported generated tests and candidate patches.

The runner is designed with multiple restrictions, including a pinned container image, non-root execution, disabled network access, a read-only container root filesystem, dropped Linux capabilities, resource limits, and execution timeouts. Sandbox execution is disabled by default and must be deliberately enabled in a controlled environment with the required image and security configuration.

The system distinguishes between a model proposing a test, a patch passing application checks, and a test actually executing successfully. When sandbox preflight checks fail or execution is unavailable, the expected result is an appropriate blocked or not-run status—not a fallback to executing repository code on the host.

These controls reduce risk; they do not eliminate every risk associated with running untrusted code or granting a controller access to the Docker daemon.

Challenges and What I Learned

The most difficult part was not getting an AI model to generate a code review. It was making the surrounding system reliable enough that its results could be evaluated and trusted appropriately.

I had to account for unsupported model claims, malformed structured output, invalid patch paths, unreliable external services, stale pull-request revisions, background-job recovery, and the risks of executing untrusted code.

This shaped the project around strict validation boundaries and explicit execution statuses. Static evidence must remain distinguishable from inference. A proposed patch must not be described as a successful repair simply because a model generated it. If execution does not happen, the report must not claim that a test passed.

Building ArchitectAgent taught me that useful AI developer tools require much more than prompting. Reliability, security, observability, reproducibility, and honest reporting are as important as the model itself.

Demonstration and Results

The recorded demonstration uses a controlled, seeded regression in which a pricing function changes from dollars to cents while an untouched checkout caller continues to interpret the returned value as dollars.

The demonstration captures the workflow from the original regression through the impact report and proposed repair to candidate-patch validation:

• The original regression test fails on the seeded bug.
• The generated candidate patch passes application validation.
• The patch is applied in a disposable Docker container.
• One targeted candidate-patch test passes in the recorded run.

The overall scan completed with warnings. The historical saved report also contains a stale limitation statement; this caveat is disclosed in the demonstration rather than hidden. The structured execution records show the baseline failure and candidate-patch test success, but one passing test does not establish that a patch is correct in every situation.

Beyond the controlled demonstration, the project's automated test suite passed 180 tests. Ruff formatting, Ruff linting, mypy checks, and the recorded GitHub Actions workflow also passed.

These results demonstrate the tested workflow on a controlled example. They are not a claim that every repository, AI finding, or proposed repair will succeed.

What's Next

ArchitectAgent is an MVP, and there is more to explore.

Future work includes expanding language and dependency coverage, evaluating the system against a broader collection of seeded regressions, improving impact-analysis precision, and strengthening the reliability of isolated test execution and repair validation.

My long-term goal is to make pull-request review more proactive: helping developers discover indirect consequences earlier, understand why a change matters, and evaluate proposed repairs with evidence before deciding what to merge.

Find the impact. Validate the proposal. Keep the developer in control.

Built With

• ai
• docker
• fastapi
• github-actions
• github-api
• nebius-token-factory
• nvidia-nemotron
• postgresql
• pytest
• python
• redis