Understudy

Understudy plays the attacker in rehearsal: it finds the path from the internet to your data in your Terraform, proves it in a sandbox, writes the fix, and proves it’s closed.

Comment

Inspiration

Every IaC scanner I've used tells me what is misconfigured — "this bucket is public," "this security group is open" — and leaves the part that actually matters to me: can an attacker reach my data, and is this finding even real? As a platform engineer I've spent too long triaging walls of "high severity" findings that turn out to be unreachable, while the one chain that genuinely connects the internet to a secret hides in the noise.

I wanted a tool that does what a red-teamer does: not list misconfigurations, but find the path from the internet to sensitive data, and then prove it — not with a model's confident guess, but with a test that actually runs. The name says the goal: Understudy plays the attacker in rehearsal, so the real one never gets the part.

What it does

Understudy reads your Terraform, builds a resource graph with evidence (file + line) on every node and edge, and finds attack paths from the internet to sensitive data — S3, Secrets Manager, RDS, DynamoDB. For each path it:

  1. Proposes the path with an LLM — but never trusts it.
  2. Verifies it against the graph — a path containing an edge that doesn't exist is rejected.
  3. Proves it in a disposable sandbox — it generates a policy test that must fail on the code as written, then applies a fix and shows the same test pass. Red → green, with the diff.

Nothing is labelled "proven" without stored red→green evidence. Repo text is treated as untrusted input, so planted prompt-injection instructions are ignored, not executed.

How we built it

  • Core loop: the LLM proposes, the graph verifies, the sandbox proves. Python 3.12 / FastAPI / Pydantic, networkx for the graph, Checkov inside the sandbox for the proofs, Next.js + React Flow for the UI.
  • Models: three NVIDIA Nemotron tiers on Nebius Token Factory, routed per step through one OpenAI-compatible endpoint — Lightning for cheap classification and summarising, Super for the reasoning-heavy path proposal and test/fix writing, Ultra to rank and narrate, with Nano as the fast-tier fallback. A full scan of the sample app is 10 calls, 34,597 tokens, $0.032 — the cheapest tier carries ~1% of cost and the top tier runs only the 2 calls that need it.
  • Proofs: Token Factory (Contree) Sandboxes — each Checkov run spins up a disposable instance from a pinned image, with the repo uploaded as files, never cloned. Copy-on-write images map exactly onto "branch per hypothesis, prove, roll back."
  • Deployed on AWS, Terraform-only: ECS Fargate (arm64), an internal ALB reached through a CloudFront VPC origin so the load balancer is never internet-facing, WAF, KMS, and a DynamoDB ledger with per-scan budget caps and a live-scan kill switch.

Challenges we ran into

  • Nemotron reasoning is on by default and billed as output. A small max_tokens returned empty answers because the whole budget went to reasoning tokens. The toggle (chat_template_kwargs.enable_thinking) is documented by NVIDIA/vLLM, not Token Factory — so I added a per-step reasoning flag.
  • Prompt caching that works but doesn't land. An isolated probe cached a repeated 4K prefix for ~79% savings, but a real 10-call scan got 0% cache hits — the rules (minimum match length, block size, write timing, TTL) aren't documented.
  • Sandboxes have no per-instance network-off switch — the biggest gap for running untrusted repo code. I compensated: upload files instead of cloning, empty env, Checkov with --skip-download, public repos only.
  • A parser bug that crashed on for_each — found by running the eval against real third-party repos, not my own sample. Fixed, tested, and redeployed to the live site.
  • Staying honest. The hardest discipline was never displaying a path the graph hadn't verified, or a fix the sandbox hadn't proven — even when the model was confident.

Accomplishments that we're proud of

  • It's live and works end to end on repos I didn't build.
  • I evaluated it against held-out repos (CloudGoat + TerraGoat), not just my own demo: 25 of 28 findings real, recall 2/3 — and I report openly that the proof loop only lands on 5 of 28 unseen paths, because an honest number is worth more than a flattering one.
  • Verified, not guessed — red→green proofs backed by real sandbox evidence.
  • Cost discipline — ~$0.03 a scan, with hard budget caps and a kill switch.
  • Built solo, on a deadline.

What we learned

  • The model is the weakest link for proving, and that's exactly why the graph and sandbox matter. The LLM is strong at proposing candidate paths and unreliable at writing a fix that passes on an unfamiliar repo (5/28). The verification layer is what makes the output trustworthy.
  • Tiered routing is real cost control — reserving the 550B model for the two calls that need it kept a full scan at three cents.
  • Held-out evaluation finds bugs your own demo never will — it surfaced both the for_each crash and a precision bug where metadata-only permissions were counted as data access.

What's next for Understudy

  • Better fix generalization — raising that 5/28 proof rate on unseen repos is the clearest next win.
  • Model containers — ECS/EKS tasks; the one documented path the eval missed was an ECS takeover.
  • More formats and clouds — CloudFormation/CDK, Kubernetes/Helm, and Azure/GCP adapters behind the same graph.
  • CI integration — a GitHub Action and PR comments, so Understudy runs on every pull request.
  • Fix the precision bug — metadata-only actions shouldn't yield data-access edges.

Built With

  • amazon-web-services
  • checkov
  • contree-sandboxes
  • fastapi
  • nebius-token-factory
  • nextjs
  • nvidia-nemotron
  • openai-api
  • pydantic
  • python
  • react-flow
  • tailwindcss
  • terraform
Share this project:

Updates

Submission history