TerraGuard

Inspiration

Any general-purpose LLM can already write decent Terraform. That was never the hard part. The hard part is the gap between "the model says this is safe" and "this just touched a real AWS account." Nobody has actually closed that gap. Today you get two broken options: paste AI-generated code into production and hope for the best, or keep AI away from infrastructure entirely and lose all the speed it promises.

We also kept running into a more specific problem: general LLMs are trained on a fixed snapshot of the internet, so they make up syntax as cloud provider APIs change over time. They also have zero visibility into a company's private modules, naming rules, or internal security policy. We wanted to find out: could retrieval-augmented generation (RAG) actually fix that? And if it did, would the result be trustworthy enough to let near a real cloud account?

What it does

TerraGuard turns a plain-English request into real, reviewed AWS infrastructure. It is not just a code generator with extra steps, it is a full safety pipeline:

Generate → Validate & Self-Heal → Guardrails → Human Review → Apply / Destroy

Self-healing generation: every output is run through real terraform validate and tflint, with up to 3 automatic repair attempts. A separate check makes sure the repair step can't cheat by quietly deleting a resource just to make the error go away.
Blast-Radius Guard: before a human ever sees the plan, we scan the real Terraform plan output for problems: touching resources it doesn't own, overly broad IAM permissions, and cost limits being exceeded. This is plain code doing the check, not a second AI just trusting the first one.
Trust Score: a single score built from how confident the retrieval was, how confident the reranker was, whether validation passed, and whether the guardrails passed. This replaces the flat, always-the-same confidence that LLMs normally show, no matter how solid or made-up the answer actually is.
Human-in-the-loop, always: a person can approve, request a small targeted fix (not a full rewrite), apply, or destroy. Nothing touches real infrastructure without an explicit human decision, and that person sees the real plan, real cost, and guardrail results before deciding.
No long-lived secrets: the agent logs into AWS using a short-lived, temporary session (STS AssumeRole with an External ID), the same secure pattern companies like Datadog and Vercel use to connect to customer AWS accounts.
Per-resource citations: hover over any generated resource to see exactly which document justified that choice. This is a real lookup, not the AI just claiming it checked something.
SRE Upload Mode: upload existing, older Terraform code and run it through the same validate, guardrail, and review pipeline, not just for new code.

How we built it

The core is a LangGraph StateGraph with SqliteSaver for saving progress on each run. This means the full state of a pipeline run, including the generated code, trust score details, and guardrail results, is saved and can sit paused indefinitely while waiting for human review.

Retrieval: We use hybrid RAG on a ChromaDB database built from the official terraform-provider-aws documentation (15,000+ chunks). A user's question is expanded into 3 related versions by Gemini (MultiQuery expansion), and the results are then re-scored by a CrossEncoder reranker (ms-marco-MiniLM) to keep only what's actually relevant before it reaches the AI's prompt. We use cosine similarity instead of Euclidean distance because it measures meaning, not just chunk length.

Generation and self-healing: Gemini 2.5 Pro powers the Architect (the code generator), the Fixer (the repair loop), and a Patcher that only rewrites the specific files a change request actually affects, instead of regenerating everything.

Real tools, on purpose: terraform validate, tflint, and Infracost (for real cost estimates, with a backup estimate if Infracost isn't available). These are the parts of the pipeline that must never be wrong, so we use real tools instead of trusting the AI to get them right.

Evaluation: a separate Gemini 2.5 Pro model, set to be fully consistent (temperature=0.0) and told to act like a certified Terraform expert, grades the generated code on a strict 1-5 scale. This is completely separate from the compiler-feedback loop that drives self-healing.

Tech stack: FastAPI + LangGraph on the backend, a Next.js/React frontend that gets live pipeline updates over Server-Sent Events, and SQLite for both job history and pipeline checkpoints.

Challenges we ran into

Basic RAG made things worse, not better. Our first version used plain vector similarity search. On our 12-scenario benchmark, it scored 2.7 out of 5 for code quality, lower than the plain LLM with no retrieval at all (3.3 out of 5). It was pulling in Terraform snippets that just happened to share keywords, which cluttered the context and confused the model. Adding the CrossEncoder reranker fixed this: the validation pass rate went from 67% to 100%. This single change is what made the whole RAG approach worth using.

A stress test with a made-up internal module. We told the agent to use a fictional internal company module (terraform-aws-acme-corp-vpc) that it had never seen documented anywhere except in our own test data. The plain LLM made up the syntax completely and burned 58,000 tokens looping through failed fixes before giving up. Basic RAG also failed (2 out of 5, same clutter problem). Our reranked RAG found the exact module schema we had provided and passed on the first try (5 out of 5), using 83% fewer tokens than the plain LLM.

The "destroy" button that silently did nothing. While testing against a real AWS account, we found that "Destroy Infrastructure" in the job history was sending the wrong ID into LangGraph's resume system. Instead of resuming the actual finished run, it quietly restarted the whole pipeline from an empty prompt, so the destroy step never actually ran. Worse, we found that even with the correct ID, a run that had already finished after "apply" has nothing left to resume, so the resume command would just hang forever. We fixed this by directly triggering the destroy step against the saved run state instead of going through the normal resume path.

Live progress wasn't actually live. Our frontend was supposed to stream pipeline progress in real time, but instead nothing updated until the whole run finished. The cause: our local dev server's proxy was fully buffering the streamed response whenever the browser asked for compressed data, which browsers always do by default. A command-line test without compression streamed fine through the same proxy, but a real browser never got any data until the connection closed. We fixed it by having the frontend talk directly to the backend in development, skipping the proxy.

Every run was re-downloading a 675MB file. Without a shared cache, terraform init was re-downloading the entire AWS provider plugin from scratch on every validate, plan, apply, and retry, sometimes 3 times in a single run. Adding a shared plugin cache folder fixed it in a few lines of code.

Accomplishments that we're proud of

100% validation pass rate with our full RAG pipeline, compared to 67% for a plain LLM, plus a perfect 5.0 out of 5 code quality score from an independent AI grader, not something we scored ourselves.
2.13x faster and 65% cheaper per run after a deep optimization pass, we traced exactly where every second and every token was being spent, node by node, and cut total time from 178 seconds down to 83.8 seconds.
A working destroy pipeline tested against a real AWS account, not a simulation, including catching and fixing a bug (a silent, do-nothing destroy button) that would genuinely be dangerous in any tool that touches production infrastructure.
A safety layer that is fully rule-based, not AI-based: every check that allows a real "apply" or "destroy" to happen reads the actual Terraform plan or calls a real tool. No step between "the AI generated this" and "this touched AWS" is just one AI trusting another AI.

What we learned

Reranking isn't a nice-to-have for RAG, it's the whole point. The gap between basic and reranked retrieval wasn't a small improvement, basic RAG was actually worse than using no retrieval at all. That's a much stronger finding than "RAG helps," and we only caught it because we benchmarked carefully instead of trusting how a demo looked.

RAG is not a cure-all, and pretending otherwise would have wasted our time. For brand-new public AWS features the model had never seen before, a plain error-and-retry loop fixed itself fine without any retrieval at all. RAG's real value is enforcing a company's own private, undocumented standards that a plain LLM has no way of knowing about. That's where we focused our effort instead of overselling what RAG does.

Trust needs to be measured, not just claimed. An LLM's own stated confidence is nearly useless as a safety signal, it sounds the same whether the answer is solid or completely made up. A Trust Score built from retrieval confidence, reranker confidence, and real validation results is the only kind of "confidence" we would actually trust to gate a real AWS action.

What's next for TerraGuard

Drift detection: catch not just infrastructure drift (what's actually deployed vs. what was last applied), but also policy drift (when a company's internal rules change after a resource was already created).
Exportable audit trail: a one-click export combining the Trust Score breakdown, the plan diff, who reviewed it, and the outcome, ready to use as change-management evidence.
Better citations: show exactly which parts of a resource came from AWS documentation vs. internal company policy, setting by setting, not just resource by resource.
Support for more cloud providers: bring the same guardrail-first approach to Azure and GCP, not just AWS.