Inspiration
AI coding tools are becoming good enough to generate infrastructure-as-code repairs, but cloud security fixes are not like ordinary code edits. A Terraform change can pass a scanner while still leaving the real cloud risk unresolved.
That gap inspired TerraProbe. We wanted to test a simple but important question:
When an AI says it fixed a Terraform security bug, did it actually fix the vulnerability, or did it only satisfy the tool checking it?
What it does
TerraProbe evaluates AI-generated Terraform security repairs using a layered oracle framework.
Instead of trusting one signal, TerraProbe checks repairs across multiple levels:
- whether the targeted scanner finding disappears;
- whether the Terraform still validates;
- whether the plan is reachable;
- whether the intended infrastructure behavior is preserved;
- whether the repair introduces new findings or side effects;
- whether a human review would consider the fix real, incomplete, destructive, or deceptive.
The goal is to expose “deceptive fixes”: repairs that look successful to an automated scanner but do not actually resolve the underlying security risk.
How we built it
We built TerraProbe around a pipeline for generating, testing, and adjudicating Terraform repairs.
First, we collected vulnerable Terraform examples and used large language models to generate candidate fixes. Then we evaluated those fixes with infrastructure and security tooling, including scanner checks, Terraform validation, and plan-level comparison. Finally, we added human review to classify whether the fixes were meaningful or misleading.
The project combines:
- LLM-generated Terraform repair attempts;
- Checkov-style security scanning;
- Terraform validation and planning;
- layered oracle checks;
- manual adjudication of repair quality;
- a taxonomy for intended fixes, invalid repairs, hallucinated additions, destructive changes, and deceptive fixes.
The research version of TerraProbe was also prepared as an arXiv preprint, so the project is both a prototype and an empirical study.
Challenges we ran into
The hardest part was that a repair can look correct from one angle and fail from another.
A scanner may say the original issue is gone, but that does not prove the cloud resource is now secure. Some fixes remove the warning by changing, deleting, or avoiding the relevant configuration instead of repairing it properly. Others introduce unrelated changes that make the system harder to trust.
Another challenge was plan reachability. Not every Terraform example can be cleanly validated or planned, especially when providers, variables, or environment assumptions are missing. That made it necessary to separate “the AI failed” from “the test environment could not fairly evaluate this case.”
The final challenge was designing a fair classification scheme. We needed categories that captured more than pass/fail, because the central issue was not just whether a tool accepted the fix, but whether the fix was actually faithful to the intended security repair.
Accomplishments that we're proud of
We are proud that TerraProbe turns a vague concern about AI-generated security fixes into something measurable.
The project shows that “scanner passed” and “security risk fixed” are not the same thing. It also gives a practical framework for testing that difference instead of relying on intuition.
We are especially proud of:
- building a layered evaluation pipeline for Terraform security repair;
- identifying cases where AI-generated fixes appear successful but remain unsafe or misleading;
- creating a repair taxonomy that captures subtle failure modes;
- connecting automated checks with human security judgment;
- producing a research artifact that can support future work in AI-assisted cloud security.
What we learned
We learned that AI repair evaluation needs stronger oracles than ordinary benchmark-style pass/fail checks.
For cloud infrastructure, a fix is not trustworthy just because a scanner no longer complains. A real fix has to preserve the intended resource, address the security risk, avoid harmful side effects, and remain deployable.
We also learned that human review is still essential for this kind of security work. Automation can narrow the problem, but it cannot always tell whether the AI repaired the vulnerability or simply changed the code in a way that hides the warning.
Most importantly, TerraProbe showed us that AI security repair should be judged by real-world risk reduction, not by whether it can satisfy a single automated tool.
What's next for TerraProbe: AI Cloud Fix Evaluator
Next, we want to turn TerraProbe into a more interactive evaluation system for AI-generated infrastructure repairs.
Future work includes:
- adding a cleaner web dashboard for viewing repair outcomes;
- supporting more Terraform security rules and cloud providers;
- expanding the dataset beyond the current study;
- comparing more models and prompting strategies;
- making the human adjudication workflow easier for security reviewers;
- integrating TerraProbe into developer workflows so teams can inspect AI-generated cloud fixes before trusting them.
Long term, TerraProbe could become a trust layer for AI-assisted DevSecOps: a system that helps teams decide when an AI-generated security fix is actually safe to use.
Log in or sign up for Devpost to join the conversation.