-
-
WOLF — AI Change Gate: Compare AI plans, detect critical risks, and explain every PROMOTE, HOLD, or REJECT decision.
Inspiration
What it does
How we built it
Inspiration
AI agents are becoming increasingly capable, but there is a serious problem: how do we know whether a new version is actually better?
A longer, more convincing response does not necessarily mean a safer or more reliable system. An AI agent might appear smarter while ignoring constraints, hiding uncertainty, or overlooking essential verification steps.
That inspired WOLF — AI Change Gate, an independent evaluation system designed around one question:
Should this AI change be promoted?
What it does
WOLF evaluates proposed improvements to AI-agent planning workflows before they are approved.
The core system:
- Gives a baseline planner and an improved BrainCore planner the same request.
- Evaluates both using five consistent criteria: ambiguity resolution, assumption exposure, constraint retention, definition of done, and verification readiness.
- Independently detects vague instructions, conflicting requirements, and attempts to manipulate the evaluation.
- Produces an explainable decision: PROMOTE, HOLD, or REJECT.
- Generates SHA-256 evidence receipts.
- Runs an eight-case regression benchmark containing both successful and blocked scenarios.
A critical feature is that WOLF can refuse promotion even when the candidate receives a high score. For example, an under-specified request can score 88/100 and still receive HOLD because essential information is missing.
WOLF also includes a separate External Evidence Lab, where users can paste structured planning outputs from different AI agents and evaluate them using the same heuristic rubric.
How we built it
WOLF was developed as a standalone web application using JavaScript, browser ES modules, Node.js, HTML, CSS, and Web Crypto.
We organized the project around a defined scope, product requirements document, and technical specification, following the Devpost Learn Skill Pack planning approach.
The architecture separates request integrity, baseline generation, candidate planning, evaluation, final decisions, and evidence generation.
Automated tests, GitHub Actions, and real-browser testing were used to verify the application.
The competition demonstration runs without requiring a paid AI API, login, or external model.
Challenges we faced
One of the biggest challenges was preventing the evaluator from rewarding superficial improvements.
An early approach could confuse longer requests with better-defined requests. We improved the Request Integrity system to consider meaningful requirements, contradictions, and missing information.
Another challenge was ensuring that a strong numerical score could not override critical safety or verification blockers.
We also deliberately reduced the scope of the competition demonstration to focus on one clear, working product.
Accomplishments we're proud of
- A functioning, publicly accessible AI change-gate application.
- Explainable PROMOTE, HOLD, and REJECT decisions.
- Five transparent evaluation metrics.
- An independent Request Integrity system.
- A fixed benchmark with four PROMOTE and four HOLD cases.
- 68 automated tests passing in the documented October 9 test run, with subsequent GitHub CI checks also passing.
- SHA-256 evidence receipts.
- A separate, browser-tested External Evidence Lab.
- A public open-source repository and a working demonstration.
What we learned
The most important lesson was that verification matters more than appearances.
Building a useful AI evaluation tool requires clear criteria, reproducible tests, explicit constraints, and evidence of failure—not just impressive positive examples.
We also learned the value of keeping a proof of concept focused. A small application that reliably demonstrates its central idea can be more useful than a complicated system with unverified capabilities.
What's next
The next goal is to evaluate genuine outputs from multiple external AI providers, develop independently labeled evaluation datasets, and measure agreement with human reviewers.
The current WOLF scoring system is deterministic and heuristic. It is not presented as a scientifically calibrated measure of general AI intelligence.
Our longer-term vision is to make independent, evidence-based AI change evaluation accessible to developers building increasingly autonomous systems.
WOLF's principle is simple: Don't promote an AI change because it looks smarter. Make it prove it.
Log in or sign up for Devpost to join the conversation.