-
-
Solution
-
Problem Statement
-
Architecture
-
Next stages
Inspiration
Every production incident we've seen with Python web apps follows the same pattern: a vendor — Stripe, Twilio, SendGrid — returns a 502 or hangs for 30 seconds, and the app crashes with an unhandled exception. The failure is discovered by a user, not a test. Not because the developers were careless, but because vendor failure modes are invisible in local development. There's no easy way to simulate "what happens if SendGrid returns malformed JSON at 2 AM."
We wanted to automate the entire loop: discover which vendors your app talks to, prove it fails under realistic failure conditions, fix the code, and validate the fix — triggered automatically on any PR open, with no human intervention in between.
What it does
GhostVendor is an autonomous vendor resilience pipeline. Open a pull request on a Python Flask or FastAPI app whose vendor clients use configurable endpoints or standard SDK clients — GhostVendor:
- Discovers vendor dependencies by scanning the source code with AST analysis, then grounds criticality scores in real-world reliability data from Tavily (known outages, SLA breach reports, documented failure patterns)
- Generates Evil Twins — stateful FastAPI mock servers that impersonate each vendor and inject five chaos modes: 30-second timeouts, 502 bursts, 429 rate limits, malformed JSON, and empty responses
- Security gates the generated code through a 3-layer check: static AST inspection, Nebius Contree sandbox execution, and LLM policy review — before any code runs in the pipeline execution environment
- Verifies resilience by pointing the real app at the Evil Twins and running a Sense→Reason→Act loop: observe baseline, plan the attack, execute chaos scenarios, score 0–100. Evil Twins run as local FastAPI processes; the target app's vendor endpoints are overridden via env vars before each chaos scenario, then restored.
- Diagnoses every failed scenario with root-cause analysis down to the specific function and line
- Patches the code, validates the patch against every previously-failed scenario in an isolated copy of the repo, and opens a ready-for-review PR only after chaos validation proves it survives
If all vendors score above threshold before patching, GhostVendor exits cleanly with no PR opened. If patching fails after 3 retry cycles, it opens a draft findings PR with the full diagnostic report. No run ends silently.
Six specialized agents run in sequence, each matched to the model that fits its job:
| Agent | Model | Role | Why this model |
|---|---|---|---|
| Vendor & Repository Detective | Nemotron Ultra | AST scan + Tavily web intel + LLM enrichment | 550B for complex multi-step reasoning: tracing criticality from env vars through source, grounding scores in real-world outage data |
| Adversarial Twin Generator | Nemotron Ultra | Generates Evil Twin FastAPI mock servers | Produces the most structurally correct FastAPI code; Nemotron-3.5-Super-VL activates automatically as fallback on rate-limit |
| Context Guard | Nemotron Nano | 3-layer security gate before any code runs locally | Speed over depth — classifies code intent in ~1s vs 8–12s for Ultra; tight structured output (SAFE/LOW/MEDIUM/HIGH/BLOCKED) suits a smaller model |
| Resilience Verifier | Nemotron Ultra | Sense→Reason→Act chaos injection + scoring loop | Multiple LLM calls per run (plan attack, decide after each scenario); Ultra's instruction fidelity prevents cascading hallucinations as context grows across 30+ scenarios |
| Runtime Debugger | Nemotron Ultra | Root-cause analysis per vendor per failed scenario | Must cite specific function names and line numbers — requires Ultra's reasoning depth to produce actionable diagnoses |
| Patch Generator | DeepSeek-V4-Pro-0813 | Full-file patch generation + local validation loop | Sends 2,000–4,000 tokens of source, expects a complete fixed file back; DeepSeek-V4-Pro-0813 consistently returns valid JSON at this scale where Ultra occasionally truncates |
All three models run through a single Nebius Token Factory endpoint under one API key.
How we built it
Orchestration: A Python state machine controls every transition — DISCOVER → ATTACK → GUARD → VERIFY → DIAGNOSE → REMEDIATE → VALIDATE. The LLM never decides what runs next. This is intentional: auditability, recoverability, and safety over autonomy. Every state is logged. Retries happen at the failed stage, not from scratch.
NVIDIA Nemotron models: We used the full tier stack matched to what each agent actually needs. Ultra (550B) handles complex multi-step reasoning — tracing vendor criticality through source code, planning chaos attack sequences, making mid-attack decisions as evidence accumulates. Nano handles the security gate where speed matters more than depth: classifying code intent in ~1 second versus 8–12 seconds for Ultra. DeepSeek-V4-Pro handles patch generation because it consistently returns valid JSON at 3,000+ token output lengths where Ultra occasionally truncates.
Nebius Token Factory: Single OpenAI-compatible endpoint (https://api.tokenfactory.nebius.com/v1/) for all five models under one API key. nebius_client.py is 80 lines. Switching a model is a one-line change. When Ultra hits a transient error, it falls back automatically to Llama-3.1-Nemotron-Ultra-253B — same endpoint, just a model name swap.
Nebius Contree Sandbox: Agent 2 generates Python code that will run on the developer's machine. Before it does, Agent 3 sends it to Contree — a Nebius-hosted isolated execution environment — and observes its actual runtime behavior. Network exfiltration, subprocess spawning, sensitive file access: caught in the cloud, not on the developer's machine. This is the only part of the pipeline not reproducible with any commodity cloud service.
Tavily: Two targeted searches per vendor — {vendor} API failures incidents reliability SLA downtime and {vendor} API timeout rate limit errors 503 502 common failures — injected into the Agent 1 prompt before criticality scoring. The LLM uses this real-world outage and SLA data to calibrate the criticality score that drives the attack order and the overall resilience score weighting. A vendor with a documented history of cascading failures ranks higher and moves the overall score needle more than one with a clean track record. Without Tavily, criticality is pure business-logic guesswork.
Trigger: Two real paths into the same pipeline:
- Dashboard (demo path): The live Streamlit dashboard button calls the GitHub Actions
workflow_dispatchAPI. A GitHub-hosted Ubuntu runner picks up the job, clones the repo, and runsmain.pyagainst the demo PR. Judges and reviewers can trigger a run directly from the browser and watch it execute in real time — no PR needed. - Automatic (real usage path): Any PR opened or reopened on a registered target repo fires a
pull_requestwebhook to a self-hosted runner. GhostVendor runs automatically against that PR — no human trigger, no dashboard.
The dashboard path exists for demos and on-demand runs. The self-hosted path is what production use looks like: open a PR, get a resilience report. Both paths run the same main.py pipeline end-to-end. A typical run completes in 8–12 minutes and consumes approximately $0.40 in Nebius token costs.
As adoption grows, the self-hosted runner moves to a job queue (SQS + ECS Fargate) with a GitHub App registering webhooks across multiple repos.
Dashboard: Streamlit with dual-backend persistence. Locally the dashboard uses SQLite. On Streamlit Cloud, it reads from Supabase (hosted Postgres) — the same database the GitHub Actions runner writes pipeline events to in real time. The D3.js neural network visualization polls Supabase every 4 seconds and lights up each agent node as the pipeline progresses.
Patch safety: Agent 6 returns the complete fixed file, never a partial diff — no NameError from missing imports. When a fix changes what a function returns (e.g. now returns {"error": ...} instead of raising), Agent 6 automatically patches every call site that does bare result["key"] access — preventing 500s in route handlers after the client is fixed. A zero-diff detector rejects patches identical to the original and forces a real retry. A JSON repair pass handles truncated LLM output — auto-closing brackets, escaping literal newlines inside strings — before giving up on a response.
Resilience scoring: vendor_score = baseline_pts (20) + scenario_score (0–80), capped at 100. overall_score is a criticality-weighted average across all vendors — higher-criticality vendors (Stripe over SendGrid, for example) move the needle more. The rescore pass after patching runs only originally-failed scenarios, so a vendor that failed 2 scenarios and fixed both scores 100, not a fraction.
Evals: A behavioral assertion suite runs the full pipeline in dry_run mode against a separate intentionally-fragile eval target — Twilio + Mailgun, vendors the pipeline was not built against. Four assertions: vendor discovery, pre-patch score below 50, non-empty patch per vendor, score improvement after patch. All 4 pass. Cost, latency, false-positive detection, and security gate recall evals are on the roadmap.
Challenges we ran into
Timeout scenarios corrupting subsequent test runs. When the Evil Twin injects a 30-second timeout, the demo app hangs with a half-open connection to the twin. The next vendor's test inherits the stuck thread. We solved this by resetting the Evil Twin process after every timeout scenario and launching a fresh app instance per vendor.
LLM-generated code safety. Agent 2 generates a full FastAPI server that runs on the developer's machine. A hallucinated twin could exfiltrate env vars or open a reverse shell. We built a 3-layer gate (AST → Contree → LLM policy) where any HIGH or BLOCKED risk discards the twin before it touches the local filesystem.
Patch validation with stale bytecode. Python caches .pyc files. Applying a patch to a temp copy of the repo and re-running the app would sometimes execute the old unpatched bytecode. We added an explicit __pycache__ purge step and PYTHONPATH remapping to force the patched source.
PYTHONPATH remapping across arbitrary repo layouts. Repos use src/, lib/, ., and other layouts. The validation step needs to remap PYTHONPATH from the original repo root to the temp patched copy without knowing the layout in advance. We made it repo-agnostic by detecting the layout from the editable install .pth files, then disabling those and injecting the correct temp path.
JSON truncation at large output sizes. Agent 6 returns the full patched source file — sometimes 3,000+ tokens of output. Ultra occasionally truncates at that scale, returning broken JSON. We added a JSON repair pass and a zero-diff detector (patches identical to the original are rejected, forcing a real retry).
Port TIME_WAIT on rapid restarts. Starting and stopping Evil Twin processes quickly leaves ports in TIME_WAIT. We added OS ephemeral port fallback so a twin that can't bind its preferred port picks a free one automatically.
Accomplishments that we're proud of
The pipeline actually works end-to-end. Open a PR on the demo app, watch GhostVendor run through all seven pipeline states (DISCOVER → ATTACK → GUARD → VERIFY → DIAGNOSE → REMEDIATE → VALIDATE), and receive a validated fix PR — with the patch proven against chaos before it's committed. That's the full loop working autonomously.
The security gate is real. Contree sandbox execution of LLM-generated code before it runs locally is not a checkbox — it's a meaningful guarantee that a hallucinated or malicious Evil Twin can't reach the developer's machine.
Evals on an unseen repo. The eval suite runs against Twilio + Mailgun — vendors the pipeline was never built against — and passes all 4 behavioral assertions. Agent 1's AST scan + Tavily enrichment generalizes to vendors it hasn't seen before.
Model tier matching is deliberate. We didn't just use the biggest model for everything. Nano on the security gate, Ultra for reasoning-heavy agents, DeepSeek for large-output patch generation — each choice has a measured reason behind it.
80-line model client for 5 models. Token Factory's single endpoint made multi-model orchestration simple enough to fit in 80 lines. That's not a demo shortcut — it's what the production client actually looks like.
Limitations
- Python only. The AST scanner, Evil Twin generator, and patch agent are built for Python Flask and FastAPI. Node.js, Go, and non-HTTP vendors (Redis, SQS, database connections) are not supported yet.
- Env-var-configurable endpoints only. GhostVendor overrides vendor base URLs via environment variables. Apps with hardcoded URLs in the source are not detected or patched.
- HTTP-level chaos only. Evil Twins simulate timeouts, 502s, 429s, malformed JSON, and empty responses. They cannot simulate database connection drops, auth token expiry, partial payload corruption, or DNS failures.
- One function patched per vendor. Agent 6 targets the single most critical function identified by the root-cause agent. Complex vendors with failures spread across multiple call sites may need manual follow-up after the automated fix.
- Rescore is not a full regression. The after-patch score reruns only the originally-failed scenarios — it does not verify that previously-passing scenarios still pass. A patch that fixes timeouts but breaks clean-response handling would not be caught.
- Evals cover one target repo. The behavioral eval suite runs against a single intentionally-fragile demo app. Generalization to arbitrary real-world repos beyond the eval target is validated only by manual test runs.
What we learned
Deterministic orchestration beats autonomous agents for a pipeline like this. We considered letting an LLM route between stages. We're glad we didn't. A Python state machine that logs every transition, retries at the exact failed stage, and always falls back to a findings PR is far more debuggable and trustworthy than an agent that might decide to skip the security gate.
Sandbox execution of LLM-generated code is non-negotiable. We didn't appreciate how important Contree was until we started testing edge cases in Evil Twin generation. LLMs hallucinate code that looks correct but behaves unexpectedly at runtime. AST analysis catches syntax-level risks; only sandbox execution catches runtime behavior.
Real-world web data changes the quality of criticality scoring. Tavily's two-query-per-vendor approach — risk profile and known failure modes — produces materially different criticality rankings than AST analysis alone. Vendors with documented outage histories rank higher, which means the pipeline attacks the most dangerous vendor first.
Validation before PR is the right bar. Proposing a patch that might fix the problem is not the same as proving it does. Building the local validation loop — isolated copy, fresh twins, re-run only failed scenarios — was the hardest engineering in the project and the most important.
What's next for GhostVendor
Stage 2 — Broader language and vendor support
- Node.js / TypeScript: AST scanner and startup detector for Express and Fastify apps
- Non-HTTP vendors: chaos modes for Redis timeouts, SQS delivery failures, database connection drops
- Hardcoded URL detection: find vendors not using env vars via string literal analysis
- Multi-endpoint vendors: attack each endpoint independently
Stage 3 — Production deployment
- GitHub App with webhooks on
pull_request.opened— filters to fire only when vendor client files change - SQS-backed async job queue with ECS Fargate tasks
- DynamoDB + S3 for full run history at scale: score before/after, patches, PR links, token cost, duration (currently Supabase Postgres shared between GitHub Actions and Streamlit Cloud)
- AWS Secrets Manager for per-tenant credential isolation
Stage 4 — Multi-tenant and observability
- Per-repo config: vendor criticality overrides, excluded paths, minimum score threshold
- Score trajectory dashboard with PR links, time-to-fix, and token cost exportable for compliance
- On-call alerting when rescore drops below threshold after a dependency upgrade
- Feedback loop: merged patches promote to golden examples for Agent 6 prompt improvement
Log in or sign up for Devpost to join the conversation.