SILENTFAIL

Find the failure nobody noticed. SILENTFAIL uses mutation testing to uncover blind spots that ordinary tests miss, explain why they survived, and suggest targeted tests to catch them.

Comment

Inspiration

A test suite can be completely green and still miss an important bug.

That question inspired me to build SILENTFAIL.

While learning software testing, I realized that writing more tests does not necessarily mean writing stronger tests. A test can pass every time while a subtle change in the code goes completely unnoticed.

SILENTFAIL is built around a simple question:

Would my tests fail if the code were subtly wrong?

Instead of only checking whether the original code passes its tests, SILENTFAIL intentionally introduces small changes into the code and checks whether the test suite can detect them.

If a changed version still passes, SILENTFAIL has found a silent failure — a weakness in the test suite that ordinary test runs may not reveal.

What I Learned

Building SILENTFAIL taught me that developer tools are not only about making something work. They also need to be reliable, deterministic, and safe.

The biggest things I learned were:

  • How mutation testing can measure the strength of a test suite.
  • Why passing tests are not the same as strong tests.
  • How to design a deterministic mutation-analysis engine.
  • How to safely execute generated and potentially untrusted code.
  • How process timeouts and output limits prevent runaway executions.
  • How workspace containment can prevent code from escaping its intended environment.
  • How Windows and POSIX systems handle processes differently.
  • How frontend, backend, and analysis-engine components communicate.
  • How regression tests can catch problems introduced during development.
  • Why testing the testing system itself is just as important as testing the application.

One of the most valuable lessons was that some of the hardest bugs were not discovered while writing the first version. They were discovered while trying to prove that the system actually worked.

That changed how I approached the project.

How I Built It

SILENTFAIL is a developer tool built around a mutation-testing pipeline.

The core workflow is:

Code + Tests
     ↓
Analyze
     ↓
Generate Mutations
     ↓
Execute Tests
     ↓
Classify Results
     ↓
Detect Survivors
     ↓
Explain Blind Spots
     ↓
Suggest Targeted Tests
     ↓
Verify 
The application uses:

React + Vite + TypeScript for the frontend
Tailwind CSS for the interface
Node.js + Express for the API
A custom mutation engine
A sandboxed execution system
Vitest for automated testing

The most important part is the mutation engine.

For example, imagine a function contains:

if (isMember) {
  return price * 0.9;
}

SILENTFAIL can create a mutation such as:

if (!isMember) {
  return price * 0.9;
}

If the existing tests still pass after this change, the mutation survives.

That tells the developer something important: the test suite may not actually check the member/non-member behavior strongly enough.

SILENTFAIL then explains the blind spot and can suggest a targeted test for the missing behavior.

The goal is not simply to generate lots of mutations. The goal is to turn surviving mutations into useful information that a developer can act on.

The Technical Challenge

The hardest part was not generating a changed line of code.

The hardest part was safely executing that changed code.

SILENTFAIL treats analyzed source code as untrusted input. A mutation-testing system cannot assume that every piece of code it executes is safe or well behaved.

The execution layer therefore includes protections such as:

Temporary isolated workspaces
Workspace-containment checks
Path traversal protection
Environment isolation
NODE_OPTIONS isolation
Execution timeouts
Output limits
Process cleanup
Windows process-tree cleanup
POSIX process-group cleanup

Cross-platform behavior became an especially important lesson.

An execution approach that works on Linux does not automatically behave the same way on Windows. During development, testing exposed Windows process-execution issues that required changes to the sandbox executor and regression tests.

I also found that a whole analysis can need its own timeout, not just each individual test process. That boundary was added and tested as well.

These problems made the project more than a UI exercise. They forced me to think about what happens when the software encounters unexpected or hostile input.

What Makes SILENTFAIL Different

Most test workflows ask:

Did my tests pass?

SILENTFAIL asks a different question:

Could my tests detect a subtle bug?

This is the key idea behind mutation testing.

A mutation intentionally makes a small change to the program. A strong test suite should notice that change and fail.

If the tests continue passing, the mutation has survived.

That creates a useful distinction:

PASSING TESTS ≠ STRONG TESTS

SILENTFAIL turns that difference into something developers can see, understand, and fix.

It is also not designed as a generic AI coding assistant.

The mutation engine and execution results are deterministic. The mutation status and mutation score come from the analysis engine rather than from an AI model.

Demo Result

The built-in deterministic demo demonstrates the complete SILENTFAIL workflow.

The process starts with a passing baseline test suite.

SILENTFAIL then generates approximately 20 mutations and executes the test suite against them.

The result identifies surviving mutations that the original tests failed to catch.

SILENTFAIL then provides explanations and targeted test suggestions.

After the suggested tests are added and verified, the demo shows the mutation score improving from approximately:

50% → 100%

This is a deterministic demonstration of the workflow, not a production benchmark.

The important part is the feedback loop:

Tests pass
   ↓
Mutation survives
   ↓
Blind spot discovered
   ↓
Targeted test suggested
   ↓
Test verified
   ↓
Mutation is caught
AI Usage

AI is optional in SILENTFAIL.

AI can assist with explanations and test suggestions, but it does not control the fundamental analysis result.

The deterministic mutation engine remains the source of truth for:

Mutation execution
Mutation status
Mutation score
Verification results

The application can also operate without an AI API key by using deterministic fallback behavior.

This was an intentional design decision because the core value of SILENTFAIL should not depend on an AI model simply saying that something looks correct.

Security & Reliability

SILENTFAIL treats source code as untrusted input.

That means the execution layer was designed around containment and controlled execution rather than assuming that submitted code will behave correctly.

The final implementation includes workspace containment, environment isolation, timeout handling, output limits, and platform-specific process cleanup.

The project also rejects unsupported .mjs inputs because the current execution runner is based on CommonJS.

These constraints make the current system more predictable and provide a clear foundation for future language and runtime support.

Challenges & Future Work

The project went through several rounds of testing and debugging before reaching the final release.

The biggest challenges were:

Making sandbox execution reliable across operating systems.
Correctly terminating processes after timeouts.
Preventing files and module resolution from escaping the temporary workspace.
Keeping mutation results deterministic.
Making the application useful even when AI is unavailable.
Testing the final application from a clean extracted archive rather than relying only on the development environment.

There are also several areas I would like to explore next:

Broader programming-language support
More mutation operators
IDE integration
CI/CD integration
Repository-scale analysis
More advanced automated test-generation strategies

These are future directions rather than features being claimed by the current version.

Why I Built It

I built SILENTFAIL because I wanted to explore a problem that is easy to overlook:

A green test suite can tell us that the tests passed.

It does not necessarily tell us that the tests were strong enough.

As a student developer, building SILENTFAIL gave me the opportunity to learn that difference by actually implementing the analysis engine, execution system, security boundaries, frontend, backend, and verification workflow — and then testing the system itself until the behavior was reliable.

The goal is not to replace developers or tell them that their tests are bad.

The goal is to help developers discover the cases they forgot to test.

Find the failure nobody noticed.

### My recommendation

**Use this version directly** in the Devpost **About the project** box.

Don't add LaTeX just because Devpost supports it. For SILENTFAIL, it won't add value; the code example and workflow diagram communicate the technical idea much better.

Built With

Share this project:

Updates

Submission history