SUPERVISOR: An AI Supervisor for AI Coding Agents
The Inspiration
AI coding agents can write code, modify entire projects, and help developers build software faster than ever. But generating code is only part of the job. Code can still contain bugs, security issues, broken integrations, or fixes that appear successful but haven't actually solved the problem.
That raised an important question: Who checks the work of an AI coding agent?
SUPERVISOR was created around this idea: coding agents should not simply generate code and move on. Their work should be inspected, potential problems should be identified, repairs should be coordinated, and the results should be verified before developers trust them.
What It Does
SUPERVISOR is an AI coding supervisor designed to review software projects, identify potential problems, support repair workflows, and verify whether changes resolve the reported issues.
Its workflow brings together several important stages:
- Inspect: Examine project files and available code context.
- Review: Look for potential bugs, security issues, and other problems.
- Coordinate repairs: Work with supported coding-agent integrations to help address findings.
- Verify: Check the results of changes and available tests instead of automatically assuming a repair succeeded.
- Report: Present findings and verification results so developers can understand what happened.
SUPERVISOR is designed to work with multiple coding environments and agents, rather than relying on a single AI coding tool.
How We Built It
The project was built as a modular Python application, with separate components for code inspection, review, context collection, security checks, test execution, repair coordination, verification, reporting, and agent integrations.
This modular architecture helps keep responsibilities separate and makes it easier to extend the system with additional languages, checks, and coding environments.
During development, we focused on a critical weakness in automated code review: a successful test command does not necessarily mean the application is correct. A test suite might not cover the affected feature, a programming language might not receive meaningful analysis, or a repair might appear successful without sufficient evidence.
These problems influenced our approach to verification and reporting. SUPERVISOR aims to distinguish between findings, attempted repairs, and results that have actually been verified.
Challenges We Faced
One of the biggest challenges was making the review process meaningful across different programming languages and project structures. Collecting a source file is not the same as understanding its behavior, and running tests is not proof that every part of an application works correctly.
Another challenge was preventing misleading success reports. A supervisor must be careful about what it claims: if the available checks cannot establish that a problem is resolved, the result should reflect that uncertainty.
Integrating different coding environments also requires handling their different workflows and capabilities without making the entire application dependent on one tool.
Addressing these challenges required examining the architecture, improving the separation between review and verification, and strengthening the way findings and results are represented.
What We Learned
Building SUPERVISOR reinforced an important lesson: automation is only useful when its results can be trusted.
A system that finds issues but cannot verify repairs is incomplete. A system that runs tests but ignores their coverage limitations can create false confidence. And a system that reports success without enough evidence can be more dangerous than one that openly reports uncertainty.
We also learned the importance of modular architecture, clear verification boundaries, and testing the supervisor itself—not just the applications it reviews.
What's Next
SUPERVISOR is a foundation for more reliable AI-assisted software development. Future improvements can expand language-specific analysis, strengthen behavioral testing, improve integrations, and make verification more precise.
Our goal is simple: help developers move from "the AI says it's fixed" to "the available evidence shows whether it's fixed."
SUPERVISOR — Don't just generate code. Check the work.
Log in or sign up for Devpost to join the conversation.