-
-
SourcePack checks whether AI-generated code changes are supported by the repository before they’re trusted
Inspiration
I kept seeing coding agents make changes that looked believable but depended on things that were not true about the repository. They would invent a dependency, file, command, or path, and another model reviewing the patch could miss the same mistake. I built SourcePack to check those claims against the repository itself. The loop is simple: generate -> check evidence -> correct -> verify.
What it does
SourcePack checks proposed code changes against local repository evidence. In the demo, the repository uses Flask, but the AI patch imports FastAPI. SourcePack catches that FastAPI is not declared and returns unsupported_dependency.
It can also catch:
- Missing files
- Unsupported commands
- Unsafe paths
- Protected files
- Malformed diffs
- Repository policy violations
A PASS does not mean the code is perfect; it means SourcePack did not find one of the unsupported repository assumptions it knows how to check.
How we built it
I designed the product, wrote the specifications, split the work into small tasks, and used Codex to implement them. After each task, I reviewed the code and tests instead of trusting the summary. When Codex misunderstood something or missed a requirement, I gave it a smaller correction task. SourcePack is mainly built with Python, Git, SQLite, HTML, CSS, and JavaScript.
Challenges we ran into
The hardest part was keeping the claim narrow. SourcePack does not prove code correctness, security, or runtime behavior; it checks whether a proposed change is supported by facts available in the repository. Other complex areas included deterministic diff parsing, stable findings, dependency resolution, safe credential handling, transaction rollback, and stopping the project from growing beyond the core scope.
Accomplishments that we're proud of
SourcePack became a working product in about a month. It now has a CLI, Git integration, baselines, dependency and command checks, policies, reports, replay, a local Workbench, an interactive demo, and an optional hosted layer. I am most proud that it solves one specific problem without pretending to solve everything.
What we learned
Repository hallucination is different from ordinary code failure. A patch can look reasonable and still depend on something that is false about the repository. I also learned that Codex works much better when the task is small, the behavior is exact, and the diff gets reviewed instead of accepting a confident summary.
What's next for SourcePack
Next, I want to test SourcePack against more real coding-agent changes and measure how often it catches unsupported repository claims. I also want to benchmark it on larger repositories. The longer-term idea is to place SourcePack directly inside a coding-agent loop so tools like Codex can check their repository assumptions before they trust or present a change.
Log in or sign up for Devpost to join the conversation.