Inspiration
Infrastructure as Code (IaC) tools like Terraform and OpenTofu make cloud infrastructure easier to manage, but testing that infrastructure properly remains a challenge.
A test suite can pass even when it fails to detect security misconfigurations, incorrect resource settings, or broken dependencies. Traditional test coverage doesn't necessarily tell developers whether their assertions can catch meaningful infrastructure failures.
That gap inspired Oasis, an open-source mutation testing framework designed to answer a simple question: How effective are your Terraform tests at catching real mistakes?
What it does
Oasis evaluates the quality of Terraform and OpenTofu test suites by deliberately introducing controlled faults, called mutants, into infrastructure configurations.
It then executes the existing tests and checks whether those faults are detected.
Key features include:
- 12 mutation operators targeting security configurations, resource dependencies, boolean conditions, and tags.
- Automated mutation testing using Terraform or OpenTofu test commands.
- Mutation scoring to measure how effectively test assertions detect faults.
- Survival analysis to identify missing test coverage and weak assertions.
- Zero-drift restoration to return the repository to its original state after testing.
- CLI tooling for previewing mutations, running tests, and exporting CSV reports.
- Cross-platform distribution through standalone executables for Windows, Linux, and macOS.
How it was built
Oasis is developed in Python, using a modular architecture.
The parser identifies suitable mutation targets in Terraform configuration files. Specialized mutation operators modify these targets, while the test runner executes the mutation-testing workflow.
The framework first establishes a passing baseline, validates generated mutations, executes the tests, and classifies the results.
A Git-based restoration mechanism ensures that changes are reverted after execution, including when errors occur.
The framework also includes a reporting system that generates structured CSV results for further analysis.
Challenges
One of the most important technical challenges in mutation testing is ensuring that generated changes are meaningful and valid.
A mutation should test the strength of an existing assertion rather than simply break the infrastructure configuration.
Another challenge is preserving the original repository state throughout repeated test executions. Oasis addresses this through its restoration mechanism.
Distinguishing between untested infrastructure resources and weak assertions is also important for producing actionable results.
Finally, evaluating real-world Terraform repositories introduces environmental challenges, particularly when cloud credentials are required to execute tests.
Accomplishments
Oasis was evaluated against 23 real-world Terraform repositories, generating 242 mutants.
Of these, 177 produced scorable results. Only 44 were caught by tests, giving an overall mutation score of 24.9%.
Among the 133 surviving mutants, 104 were classified as having no coverage and 29 as involving weak assertions.
These results demonstrate how passing infrastructure tests can still overlook meaningful configuration faults.
What we learned
The project demonstrates why test quality should be measured by the failures a test suite can detect, rather than simply whether tests execute successfully.
It also highlights the importance of reliable state restoration, meaningful fault classification, and reproducible evaluation when building developer tooling.
What's next
Potential future improvements include broader mutation operator coverage, additional provider-specific testing scenarios, richer reporting and visualization, and deeper integration with CI/CD workflows.
The long-term goal is to make mutation testing a practical part of Infrastructure-as-Code quality assurance.
