AI Engineering Change Auditor

An AI-powered engineering auditor that investigates code changes, reproduces bugs, proposes fixes, and verifies them with real tests before they reach production.

Comment

AI Engineering Change Auditor

Inspiration

AI can generate code in seconds, but generating code and generating correct code are two very different things.

A code change can look convincing, pass a superficial review, and still introduce a subtle bug. Developers need more than an AI assistant that points out suspicious lines or recommends a possible solution. They need a way to investigate defects, reproduce them, and verify that proposed fixes actually work.

This inspired us to explore a different approach to AI-assisted software engineering: What if AI could investigate a code change like an engineer, test its suspicions, and provide evidence before recommending a fix?

That question led to AI Engineering Change Auditor.

What It Does

AI Engineering Change Auditor is an AI-powered engineering audit system designed to investigate code changes, reproduce suspected defects, propose corrections, and verify those corrections through real test execution.

Instead of treating an AI-generated explanation as proof, the system emphasizes reproducibility, test results, and human oversight.

The intended workflow is straightforward:

  1. Investigate: Analyze a code change and identify potentially problematic behavior.
  2. Reproduce: Generate or select tests that check the suspected defect against the original implementation.
  3. Propose: Suggest a minimal correction supported by an explanation.
  4. Verify: Execute tests against the proposed change and check for regressions.
  5. Report: Record findings, proposed fixes, test outcomes, and supporting evidence.

The objective is not to eliminate developers from the review process. It is to help them make better-informed decisions with less manual investigation.

How We Built It

We designed the project around a structured, evidence-driven workflow rather than a single AI prompt that produces a code review.

The architecture separates the application interface, backend orchestration, repository integration, agent responsibilities, test execution, and audit reporting.

The planned technology stack includes:

  • Frontend: Next.js, TypeScript, and Tailwind CSS.
  • Backend: Python and FastAPI.
  • AI and agentic workflows: GitLab Duo Agent Platform.
  • Testing: Pytest and automated test execution.
  • Integration and automation: GitLab APIs and GitLab CI/CD.
  • Persistence: PostgreSQL for audit sessions, findings, test results, and reports.
  • Isolation: Controlled execution environments for testing untrusted repository code.

The GitLab Duo Agent Platform is central to our approach to agentic engineering. The aim is to give the workflow distinct investigative and verification responsibilities while maintaining traceability across the process.

We also designed the workflow around human approval. A proposed patch should not automatically be treated as safe, and the system should not silently push changes into a production branch.

The final submission should identify which components are implemented, which integrations are functional, and which capabilities remain planned.

A Simple Example

Consider a function that calculates the final price after a discount:

def calculate_final_price(price, discount_percent):
    return price * discount_percent / 100

For a price of ₹1,000 and a discount of 20%, the expected final price is ₹800. However, the function returns ₹200 because it calculates the discount amount rather than subtracting that amount from the original price.

An evidence-driven audit should:

  • Identify the suspicious calculation.
  • Establish the expected behavior.
  • Generate a test that reproduces the defect.
  • Demonstrate the test failure against the original implementation.
  • Propose a correction.
  • Run the test against the corrected implementation.
  • Execute relevant regression tests and record their actual outcomes.

This example captures our central principle: a plausible explanation is useful, but a reproducible failure and a verified correction are stronger evidence.

What We Learned

Building this project reinforced several important lessons about agentic software engineering.

1. AI confidence is not verification.
An AI-generated finding can be incorrect, incomplete, or based on an incorrect assumption. Independent tests provide a way to evaluate claims against observable behavior.

2. Reproducibility matters.
A useful audit should identify the code revision being examined, the tests executed, and the outcomes observed. Without that context, it becomes difficult to reproduce or trust a result.

3. Human oversight is a design decision.
Automation is valuable, but proposed code changes should remain reviewable. Explicit approval and isolated execution help prevent an experimental fix from becoming an unintended production change.

4. Agent orchestration requires clear responsibilities.
Separating investigation, repair design, and verification makes the workflow easier to reason about and evaluate than relying on one unrestricted agent to do everything.

5. Honest reporting is essential.
A test that was never executed must not be reported as passed. A suggested fix must not be described as verified until the relevant checks have actually run.

Challenges We Faced

One of the biggest challenges is distinguishing a genuine defect from code that merely looks suspicious. Static analysis and AI reasoning can identify candidates, but determining whether a finding represents a real behavioral problem often requires a meaningful test case.

Another challenge is safely executing code from repositories. Repository code and dependencies must be treated as untrusted. Test execution therefore needs appropriate isolation, resource limits, controlled access, and cleanup.

We also had to consider the reliability of proposed patches. A correction may resolve one failure while introducing another, which is why targeted tests alone are not always sufficient. Regression testing and transparent reporting are important parts of the workflow.

Finally, integrating agentic reasoning into a reliable engineering process requires more than connecting a model to an interface. The system must preserve the relationship between the original code, the finding, the proposed patch, the executed tests, and the final result.

These challenges shaped our focus on evidence, controlled automation, and verifiable outcomes.

What's Next?

Our longer-term vision is to make AI-assisted code review more systematic and measurable.

Potential future improvements include:

  • Deeper integration with GitLab merge requests and CI pipelines.
  • Expanded support for programming languages and test frameworks.
  • Larger reproducible benchmarks for defect detection and repair.
  • Better detection of regressions and false positives.
  • Audit history and comparisons across code revisions.
  • Evaluation of agent performance, execution cost, and verification success.

We want to evaluate the system not just by how many issues it identifies, but by how many findings are reproducible, how often proposed fixes pass independent tests, and how reliably the workflow avoids introducing new failures.

Our Vision

AI should not ask developers to trust it blindly.

It should help them investigate, test, verify, and understand.

AI Engineering Change Auditor brings that philosophy to software engineering: investigate the change, reproduce the problem, verify the fix, and preserve the evidence.

Built With

Share this project:

Updates

Submission history