Inspiration
I work with AI-assisted development a lot, and one problem kept bothering me: AI can produce code very quickly, but it can also say that something is complete without giving enough proof.
As the person defining the process, requirements, and governance of the systems we are building, I did not want to rely only on statements like “the feature is done” or “the tests are covered.” I wanted a simple way to ask:
Where exactly is the evidence inside the repository?
That became the idea behind WeAxes CanonGate.
The principle is simple:
No evidence, no certification.
CanonGate was built to help developers and reviewers check whether a software repository actually contains evidence that specific requirements were implemented.
What it does
CanonGate reviews a small source-code repository against a list of acceptance requirements.
The user can upload a ZIP file or use the included BEFORE and AFTER sample repositories.
For every requirement, CanonGate returns one of three results:
- PASS when the repository contains evidence supporting the requirement
- FAIL when the repository contains evidence showing that the requirement is not satisfied
- NOT VERIFIED when there is not enough evidence to make an honest decision
GPT-5.6 performs the semantic assessment, but CanonGate does not simply trust the model’s answer.
The model must provide the file path, line range, and exact source snippet that supports its conclusion. CanonGate then checks those citations against the uploaded repository.
When a PASS or FAIL does not have valid supporting evidence, CanonGate changes the final result to NOT VERIFIED.
The user can also download the completed assessment as JSON or Markdown.
How we built it
I approached the project from a process and governance perspective.
Before development started, I defined the main decision law:
- PASS must have positive evidence
- FAIL must have direct contradictory evidence
- Missing or unclear evidence should result in NOT VERIFIED
- Test source can prove that a test exists, but it cannot prove that the test was executed or passed
- CanonGate must never execute uploaded repository code
Codex was the principal implementation agent. It helped build the Python and Streamlit application, repository ingestion, structured GPT-5.6 assessment, citation validation, sample repositories, report generation, automated tests, GitHub Actions workflow, documentation, and deployment preparation.
My role was to define the product purpose, limits, decision rules, acceptance requirements, and review boundaries, and then scrutinize the implementation evidence before accepting the build.
The application uses:
- Python 3.12
- Streamlit
- OpenAI Responses API
- GPT-5.6
- Pydantic structured outputs
- Pytest
- Ruff
- Mypy
- GitHub Actions
- Streamlit Community Cloud
The final workflow is:
- Validate and safely inspect the uploaded ZIP
- Build a bounded in-memory index of supported source files
- Redact likely secrets before sending repository text to the API
- Ask GPT-5.6 to assess every requirement
- Validate every cited file, line range, and exact snippet
- Produce the final PASS, FAIL, or NOT VERIFIED result
- Export the report as JSON or Markdown
Challenges we ran into
The hardest part was not making the model produce an answer. The harder problem was deciding when the answer should be trusted.
We had to clearly separate:
- AI reasoning from evidence validation
- Documentation from implementation
- Test source from test execution
- Missing proof from proven failure
- The model’s proposed verdict from CanonGate’s final verdict
For example, when the BEFORE sample does not contain tests for invalid invoice amounts, CanonGate returns NOT VERIFIED instead of automatically calling it a failure. The absence of a test does not prove that the entire application is defective. It only proves that the required test evidence is missing.
We also had to put strict boundaries around uploaded repositories. CanonGate limits archive size, rejects unsafe paths and unsupported file types, ignores sensitive locations, and does not execute uploaded code.
Another challenge came during deployment. Streamlit Community Cloud initially tried to install the project through the repository configuration and failed. We traced the issue to dependency installation and added a minimal root-level requirements.txt. After that correction, the application deployed successfully.
Accomplishments that we're proud of
I am proud that CanonGate works as a real public application and not only as a concept.
The live BEFORE assessment returns:
- 2 PASS
- 1 FAIL
- 1 NOT VERIFIED
The corrected AFTER assessment returns:
- 4 PASS
- 0 FAIL
- 0 NOT VERIFIED
The project also includes:
- Deterministic validation of file paths, line ranges, and exact snippets
- Separate model verdicts and final CanonGate verdicts
- Safe bounded repository ingestion
- Likely-secret redaction
- Synthetic BEFORE and AFTER examples
- JSON and Markdown report downloads
- Automated offline tests
- Type checking and code-quality checks
- GitHub Actions continuous integration
- A public Streamlit deployment
- A complete demo that judges can run without installing the project
I am also proud that the project remains honest about its limitations. CanonGate does not claim that source inspection proves runtime behavior, and it does not claim that tests ran simply because test files exist.
What we learned
The biggest lesson was that AI-generated software needs more than another confident AI opinion.
It needs explicit requirements, visible uncertainty, traceable evidence, and independent checks.
I also learned that evidence validation and semantic reasoning are different responsibilities. CanonGate can confirm that a cited snippet really exists, but GPT-5.6 is still responsible for interpreting what that source means.
That distinction is important because a valid citation can still be misunderstood.
The project also strengthened my view that AI should help people make better decisions without silently taking over their authority.
CanonGate provides evidence. It does not authorize merges, deployments, publishing, spending, or production operations.
What's next for WeAxes CanonGate
The next step is to test CanonGate as an evidence-review gate in larger AI-assisted development projects.
I plan to pilot it first in TAYRA, then WMI, and later WTI after the repository boundaries are properly mapped.
Possible future improvements include:
- GitHub and pull-request integration
- Trusted CI and test-result evidence
- Signed build and test records
- Persistent assessment history
- Larger repository support
- Organization-specific requirement templates
- Multi-stage human approval workflows
- Security and compliance assessment packs
The long-term objective is not to replace developers, reviewers, or system architects.
It is to make AI-assisted development more accountable by requiring every important conclusion to answer one basic question:
Where is the proof?
No evidence, no certification.
Log in or sign up for Devpost to join the conversation.