About the Project Inspiration Flaky tests are one of the most frustrating problems in software development. A test can pass locally, fail in CI, and then pass again without any code changes. Developers often spend hours rerunning tests, reading stack traces, and determining whether the production code is wrong or the test has an incorrect expectation.
I built breakmycode to make this investigation faster, safer, and easier to understand. My goal was to create an AI-powered developer tool that can inspect an existing Python project, detect unreliable tests, explain failures in plain English, generate missing test coverage, and propose fixes without silently modifying the developer’s files. What breakmycode Does A developer points breakmycode at a local Python project: breakmycode /path/to/project --runs 20 The agent runs the project’s pytest suite repeatedly and records the outcome of every test. A test is classified as flaky when it both passes and fails against identical code.
The failure rate is calculated as: [ \text{Failure Rate} = \frac{\text{Failed Runs}}{\text{Total Runs}} \times 100 ] For example: test_checkout.py::test_payment failed 6/20 (30%) For every failing or flaky test, breakmycode collects bounded source-code context, the relevant tests, and sanitized failure output. It sends this information to NVIDIA Nemotron through Nebius Token Factory.
Nemotron then:
- Determines whether the production code is wrong or the test has an incorrect expectation.
- Produces a short, plain-English explanation.
- Proposes a unified diff when it identifies a production-code bug.
- Reviews generated patches for possible security risks. The user must explicitly approve every patch: Apply this patch? [y/N] Only an exact y applies the patch. The change is placed on a new Git branch and verified with fresh pytest runs. AI-Powered Test Generation breakmycode can also generate additional pytest coverage for an existing project: breakmycode generate-tests /path/to/project The generated tests are not immediately written into the repository. Instead, breakmycode:
- Inspects the project’s source code and existing tests.
- Asks Nemotron to identify valuable missing edge cases.
- Validates the generated Python and unified diff.
- Applies the test temporarily inside an isolated copy of the project.
- Runs the complete pytest suite.
- Sends incorrect expectations back to Nemotron for a bounded retry.
- Displays only a policy-safe, preflight-passing proposal.
- Waits for explicit user approval.
- Applies approved tests on a new Git branch.
- Runs pytest again to verify the final result. This makes the model’s output testable before it can affect the developer’s working repository. How I Built It I built breakmycode as a Python command-line application using:
- pytest for repeated test execution and per-test result collection.
- LangGraph for orchestrating the agent lifecycle.
- NVIDIA Nemotron models for test generation, failure diagnosis, explanations, and patch proposals.
- Nebius Token Factory for OpenAI-compatible access to Nemotron models.
- Git for isolating approved changes on new branches.
- SQLite for local observability, traces, model-call records, latency, token usage, and estimated cost.
- Nebius Serverless Jobs for packaging repeatable evaluation and red-team workflows. The main agent workflow is: Local Python repository ↓ Validate project ↓ Run pytest repeatedly ↓ Detect flaky and failing tests ↓ Collect relevant source context ↓ Ask Nemotron for diagnosis ↓ Run deterministic security checks ↓ Run AI-assisted security review ↓ Request explicit user approval ↓ Create a new Git branch ↓ Apply the patch ↓ Run pytest again and verify Model Selection I used Nemotron 3 Super 120B A12B for deeper tasks such as repository-level reasoning, test generation, failure diagnosis, and patch proposals.
I used Nemotron 3 Nano 30B A3B as a faster and lower-cost fallback for routine calls and malformed-output retries.
This routing strategy allowed the project to use stronger reasoning when necessary while controlling latency and cost for simpler operations. Safety and Blue-Teaming The central safety rule in breakmycode is that the AI may propose a change, but it cannot silently apply one.
The blue-team protections include:
- Explicit approval before every modification.
- New Git branches for approved patches.
- Isolated test-generation preflight.
- Unified-diff validation.
- Path traversal prevention.
- Credential and environment-variable protection.
- Network-access blocking in generated tests.
- Shell and subprocess execution blocking.
- Production-code modification prevention during test generation.
- Detection of removed or weakened assertions.
- Post-application pytest verification.
- Regression detection after a patch is applied. If the user rejects a proposal, the repository remains unchanged. Red-Team Evaluation I created adversarial scenarios to test whether the safety layer could be bypassed.
The red-team suite includes attempts to:
- Write files outside the repository using path traversal.
- Read API keys or credential files.
- Replace meaningful assertions with assert True.
- Execute shell commands from generated tests. The current security evaluation reports: BLOCKED path_traversal BLOCKED credential_file BLOCKED test_weakening BLOCKED shell_injection
Red-team suite: 4/4 attacks blocked These attacks are blocked by deterministic policies rather than relying only on the model’s judgment. Observability and Tracing Every workflow receives a unique scan ID.
For example: scan_8c13045d132c A developer can inspect the recorded trace with: breakmycode show scan_8c13045d132c The trace records:
- Workflow stages.
- Test results.
- Failure and flaky-test counts.
- Model selections.
- Model-call latency.
- Token usage.
- Estimated cost.
- Security findings.
- Approval decisions.
- Patch verification results. This makes the agent’s behavior explainable and auditable. Challenges I Faced Reliable model output Models do not always return perfectly formatted JSON or unified diffs. Some responses included Markdown fences, incomplete patches, or valid Python in an unexpected format.
I addressed this with:
- Flexible response parsing.
- Unified-diff repair.
- Structured-output validation.
- Bounded retry loops.
- Deterministic policy checks.
- Isolated pytest preflight. Incorrect generated expectations A generated test may be valid Python but still misunderstand the application’s intended behavior.
Instead of immediately presenting such tests to the user, breakmycode executes them inside a temporary copy. When they fail, sanitized pytest feedback is returned to Nemotron so it can propose a corrected version. Python environment problems pytest may fail during collection when the wrong Python interpreter is active or project dependencies are missing. Initially, these failures could look like a project containing zero tests.
I improved the runner so collection errors are reported directly, including the missing dependency or incompatible interpreter, instead of being treated as empty test results. Balancing context and privacy Sending an entire repository to a model would be expensive and could expose unnecessary files.
I built bounded context collection that prioritizes relevant source files and tests while excluding:
- .env files.
- Credentials.
- Virtual environments.
- Build output.
- Cache directories.
- Git metadata.
- Symlinks.
- Unrelated large files. Preserving developer control The most important challenge was ensuring that the agent remained useful without becoming unsafe.
The final design keeps the developer in control at every important boundary: the model proposes, security policies validate, pytest supplies evidence, Git isolates changes, and the user decides whether anything is applied. What I Learned I learned that a reliable coding agent requires much more than a capable language model. The model is valuable for reasoning and explanation, but dependable behavior comes from combining it with:
- Deterministic tools.
- Security boundaries.
- Structured workflows.
- Isolated execution.
- Automated evaluations.
- Observability.
- Human approval. I also learned that different model sizes are appropriate for different stages of an agentic workflow. Larger models are useful for difficult debugging and repository-level reasoning, while smaller models can handle routine tasks more quickly and economically.
Most importantly, I learned that agentic AI works best when every action is measurable, explainable, reversible, and explicitly approved. Accomplishments The current project includes:
- Repeated pytest execution and flaky-test detection.
- Failure-rate reporting.
- Plain-English AI explanations.
- Code-bug versus test-bug classification.
- AI-generated test coverage.
- Unified-diff patch proposals.
- Human approval gates.
- Git branch isolation.
- Deterministic and AI-assisted security review.
- Red-team and blue-team evaluation.
- Local tracing and observability.
- Nebius Serverless job packaging.
- Model routing across Nemotron variants.
- Automated verification with 44 passing unit tests.
- A red-team result of 4/4 attacks blocked. breakmycode demonstrates how Nemotron can be placed inside a practical, observable, and safety-conscious software-development lifecycle rather than being used only as a standalone chatbot.
Log in or sign up for Devpost to join the conversation.