CD-AVC

• 

  sample

• 

  Verification_flow

Inspiration

We kept seeing coding models described by benchmark scores, sometimes around 70 on SWE-bench. Those scores are useful, but they don’t show everything that happens when you use a model in a real codebase. The model still has to understand the task, edit the right files, satisfy the project’s requirements, and avoid breaking existing behavior.

We wanted to test a practical idea: could a small local coding model do more useful work when its changes were bounded and checked by a deterministic verification process? CD-AVC grew from that question.

What it does

CD-AVC runs a coding task through a series of explicit gates. A developer provides the task, the files the model may change, and the tests that define success. CD-AVC asks a local model for a patch, checks that the patch stays within the allowed files and meets the task contract, then runs targeted and regression tests.

If a candidate fails, the system can give the model feedback and allow a bounded retry. Each run produces an execution log and report, so developers can inspect what happened and why a change passed or failed.

How we built it

We built CD-AVC in Python with no runtime dependencies. It supports an offline mock provider for development and testing, as well as an OpenAI-compatible endpoint for local model servers.

The verification flow checks patch boundaries, Python syntax, required symbols, task requirements, and test results. Candidate changes run in a temporary workspace, and CD-AVC applies a change to the project only after it clears the configured gates. We also built a benchmark runner to compare plain model responses, retries, and CD-AVC’s verification flow on the same coding tasks.

Challenges we ran into

A model can produce a patch that looks plausible and still fails the task. Even passing a targeted test may not be enough: our benchmark runs included candidates that passed the public test but failed the hidden evaluator. That made it important to distinguish “the model returned code” from “the change met the task’s requirements.”

We also had to handle imperfect model output. Local models sometimes wrap answers in Markdown, add prose around structured data, or return malformed JSON. We made the parser tolerant of common formatting errors while keeping the file and verification checks strict.

Fair comparisons were another challenge. We needed to give each approach the same tasks and model, track retries and sampling, and report measured results without treating a small experiment as proof of broad improvement.

Accomplishments that we're proud of

We built a working end-to-end flow that can take a task from a local model through patch checks, targeted tests, regression tests, and a recorded report. The offline mock provider lets us exercise that flow without needing a model server or network access.

We also ran a real smoke test with a cached local model and confirmed that CD-AVC could accept a correct change after its verification gates passed. That result is deliberately narrow: it confirms the path works on a small task, not that the model will solve arbitrary programming work.

What we learned

A benchmark score measures performance under a particular evaluation setup. It doesn’t answer every question a developer faces when applying a model to a codebase. In our experiments, retries and additional candidate samples changed pass rates, while verification caught failures that a single visible test would have missed.

We also learned that the verifier matters as much as the model workflow. Weak or incomplete tests can approve a patch that still violates the intended behavior. A useful system needs explicit task requirements, meaningful tests, clear limits on model edits, and reports that make failures understandable.

What's next for CD-AVC

We want to test CD-AVC on a wider range of repository-level tasks and local models, with stronger task contracts and more complete test coverage. We also want to improve the benchmark process so it can better explain when verification helps, when retries help, and where the current checks still miss errors.

The longer-term goal is to make local coding-model experiments more reproducible and their results easier to interpret, one verified change at a time.

Built With

• ai
• css
• docker
• git
• gsap
• html
• javascript
• lenis
• llama.cpp
• llms
• local
• openai-compatible
• python
• three.js