GhostVendor

GhostVendor haunts your dependencies. Duo analyses your MR for vendor risks, six agents impersonate them and patch what breaks, then Duo reviews the fix before it merges.

Comment

Inspiration

Every production incident we've seen with Python web apps largely follow the similar pattern: a vendor — Stripe, Twilio, SendGrid — returns a 502 or hangs for 30 seconds, and the app crashes with an unhandled exception. The failure is discovered by a user, not a test. Not because the developers were careless, but because vendor failure modes are invisible in local development. There's no easy way to simulate "what happens if SendGrid returns malformed JSON at 2 AM."

We wanted to automate the entire loop — discover which vendors your app talks to, prove it fails under realistic failure conditions, fix the code, and validate the fix — triggered automatically on any MR open, with no human intervention in between.

But beyond the technical loop, we wanted to build something that genuinely used GitLab's DevSecOps lifecycle rather than just running a script in CI. The Duo Agent Platform gave us a way to bring real AI analysis directly into the MR thread — before the pipeline even runs, and again after it completes — turning GitLab itself into an active participant in the remediation cycle, not just a runner host. The result is a Supervised automation: the developer opens the MR and assigns the Duo reviewer — everything after that runs autonomously, and the developer reviews only the outcome: the fix MR with Duo's verdict already on record, ready to merge or reject.

Built for GitLab — Path B

GhostVendor started as our own open-source Python project on GitHub, built for the NVIDIA Nebius hackathon, using GitHub Actions and NVIDIA Nemotron models via Nebius Token Factory. For this hackathon we migrated it to GitLab, rebuilt the entire inference layer on Qwen3-235B and DeepSeek v3.2 via AWS Bedrock Mantle, and built everything that makes it a GitLab-native project from scratch: the two Duo Agent flows, the GitLab CI pipeline and downstream trigger, the GitLab MR API integration (fix MR creation, reviewer assignment, MR comment posting), the Supabase backend shared between the CI runner and the dashboard, the Duo Insights dashboard tab, and the Render deployment. The core 6-agent pipeline logic carried over; the GitLab integration — every line of it — is new work built after October 5th.

Original project (MIT licensed): github.com/NithinBR-AI/ghostvendor

GitLab project (this submission): gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43058204/ghostvendor

Deployed project (Path B): ghostvendor.onrender.com — GhostVendor's live control plane, deployed on Render. This is the public deployment of the automation itself, not a static mockup. During an active run, the GitLab CI runner writes every agent state transition to Supabase and the dashboard's D3.js agent graph polls every 4 seconds, lighting up each hex node as its agent activates — so a developer can watch DISCOVER → ATTACK → GUARD → VERIFY → DIAGNOSE → REMEDIATE → VALIDATE happen in real time from inside GitLab (the pipeline auto-posts the dashboard link as an MR comment at run start). Between runs, the Run History tab holds 15+ completed runs with before/after resilience scores, timestamps, and links to the dev MR, fix MR, and both Duo reviews each run produced. The Duo Insights tab pulls Flow 1 and Flow 2 findings live from the GitLab Notes API. No login required. Note: the dashboard is on Render's free tier — it may take 30–60 seconds to wake up on first load, but it will spin up.

The target app lives on the feature branch of each MR — that's where the chaos testing runs. The deployed project at ghostvendor.onrender.com is the pipeline's control plane and run history.

What it does

GhostVendor is an autonomous vendor resilience pipeline. Open a merge request on a Python Flask or FastAPI app — GhostVendor:

  1. Discovers vendor dependencies by scanning the source code with AST analysis, then grounds criticality scores in real-world reliability data from Tavily (known outages, SLA breach reports, documented failure patterns)
  2. Generates Evil Twins — stateful FastAPI mock servers that impersonate each vendor and inject five chaos modes: 30-second timeouts, 502 bursts, 429 rate limits, malformed JSON, and empty responses
  3. Security gates the generated code through a 3-layer check: static AST inspection, Nebius Contree sandbox execution, and LLM policy review — before any code runs in the pipeline execution environment
  4. Verifies resilience by pointing the real app at the Evil Twins and running a Sense→Reason→Act loop: observe baseline, plan the attack, execute chaos scenarios, score 0–100
  5. Diagnoses every failed scenario with root-cause analysis down to the specific function and line
  6. Patches the code, validates the patch against every previously-failed scenario in an isolated copy of the repo, and opens a ready-for-review fix MR only after chaos validation proves it survives

Two GitLab Duo Agent flows bookend the entire pipeline:

  • Flow 1 (Threat Detection): When the developer assigns @ai-ghostvendor-gitlab-ai-hackathon as reviewer on the dev MR, Duo immediately reads the diff, pulls the affected vendor files, identifies risks by severity (CRITICAL / HIGH / MEDIUM / LOW), and posts a full threat analysis inline on the MR — within ~30 seconds, before any CI job completes.
  • Flow 2 (Patch Review): When GhostVendor opens the fix MR, it automatically assigns @ai-ghostvendor-review-gitlab-ai-hackathon as reviewer. Duo reads the full patched files, validates every resilience pattern (timeouts, retry logic, circuit breakers, no silent false-success), and submits a formal MR review with an APPROVED / NEEDS REVISION verdict and per-file inline assessment.

Both flows run on claude-sonnet-5-5-vertex via the GitLab Duo Agent Platform.

Six specialized agents run in sequence, each matched to the model that fits its job:

Agent Model Role Why this model
Vendor & Repository Detective Qwen3-235B AST scan + Tavily web intel + LLM enrichment 235B MoE for complex multi-step reasoning: tracing criticality from env vars through source, grounding scores in real-world outage data
Adversarial Twin Generator Qwen3-235B Generates Evil Twin FastAPI mock servers with 5 chaos modes per vendor Produces the most structurally correct FastAPI code; Nemotron Super activates automatically as fallback on transient error
Context Guard DeepSeek v3.2 3-layer security gate before any generated code runs locally Architecturally separate from Qwen3 (which generates the twins) — eliminates same-model blind spots in the security review
Resilience Verifier Qwen3-235B Sense→Reason→Act chaos injection + scoring loop Multiple LLM calls per run; Qwen3's instruction fidelity prevents cascading hallucinations as context grows across 30+ scenarios
Runtime Debugger Qwen3-235B Root-cause analysis per vendor per failed scenario Must cite specific function names and line numbers — requires the reasoning depth of a large MoE
Patch Generator DeepSeek v3.2 Full-file patch generation + local validation loop Consistent large-output structured JSON at 3,000+ token responses; same-model choice for both the security gate and the fix side of the pipeline is deliberate

How it covers the GitLab DevSecOps lifecycle

This is not a project that touches one corner of the lifecycle. GhostVendor reaches 7 of 9 post-code DevSecOps stages through genuine, non-trivial automation:

Stage What GhostVendor does Evidence
plan Tavily searches real-world outage and SLA data per vendor; Agent 1 builds a vendor_spec with criticality-weighted attack ordering — the pipeline plans which vendors to attack first and why based on actual incident history, not guesswork Agent 1 vendor_spec JSON with criticality scores and Tavily citations
create Agent 6 opens a validated fix MR with real code changes on a new branch (ghostvendor/resilience/<timestamp>), complete with score before/after, chaos scenario results, and a link back to the triggering MR Fix MR on ghostvendor-demo-app with diff, description, and MR metadata
verify GitLab CI runs the full 6-agent pipeline as the functional verification suite. Evil Twins prove the app under real failure modes — this is not a unit test or a mock assertion, it is the actual app running against chaos servers Visible CI pipeline history in the ghostvendor project
secure Two independent security layers: (1) Agent 3 runs a 3-layer gate — AST inspection, Nebius Contree sandbox execution, LLM policy review — on every Evil Twin before it runs locally. (2) Duo Flow 1 posts a CRITICAL/HIGH/MEDIUM/LOW threat analysis inline on the dev MR before the pipeline even starts Contree sandbox logs; Duo bot comment on dev MR
release GhostVendor opens and configures the fix MR: assigns the Duo reviewer, posts resilience score before/after, links back to the triggering MR, and marks the branch for merge. The release artifact is proven — not proposed Fix MR on ghostvendor-demo-app ready to merge
monitor Real-time dashboard at ghostvendor.onrender.com: D3.js agent visualization polls every 4 seconds during a run, lighting up each agent node as it activates. Duo Insights tab fetches Flow 1 and Flow 2 findings live from the GitLab Notes API. Run History tab tracks resilience score before/after across all runs with MR links and timestamps. The pipeline auto-posts the dashboard link as an MR comment at the start of every run — the developer can watch agents activate in real time from the MR thread itself Live dashboard; MR comment with dashboard link posted at pipeline start
govern Duo Flow 2 submits a formal MR review on record in GitLab — an AI audit trail of the threat analysis (Flow 1) and the patch review (Flow 2) before any human touches the diff. The fix MR cannot be merged without this review on record Duo bot review comment on fix MR with APPROVED / NEEDS REVISION verdict

Stages not covered: package (Evil Twins are ephemeral processes, not published artifacts) and configure (GhostVendor uses CI variables but does not manage environment configuration as a lifecycle stage).

How we built it

GitLab Duo Agent Platform

GhostVendor uses two Duo Agent flows registered on ghostvendor-demo-app:

Flow 1 — Threat Detection (ghostvendor-detect.yaml) Triggers when @ai-ghostvendor-gitlab-ai-hackathon is assigned as a reviewer on the dev MR. The flow calls list_merge_request_diffs to read the diff, get_repository_files to pull the affected vendor source files, sends both to claude-sonnet-5-5-vertex for risk analysis, and posts the structured threat report via create_note. The entire flow completes in ~30 seconds — the developer has expert AI threat analysis in the MR thread before any CI job finishes.

Flow 2 — Patch Review (ghostvendor-review.yaml) Triggers when @ai-ghostvendor-review-gitlab-ai-hackathon is assigned as reviewer on the fix MR (GhostVendor does this automatically at the end of the pipeline). The flow reads the full patched files, validates every resilience pattern — timeouts, retry logic, exponential backoff, circuit breakers, no silent false-success on non-2xx, no retry exhaustion fallthrough — and calls submit_mr_review with a formal verdict and per-file inline assessment.

Why two flows: The ai_workflows OAuth scope allows reads and MR review writes but not CI pipeline triggers. Rather than trying to work around this constraint, we designed around it: two event-driven flows that each do exactly what Duo can do well — expert threat analysis at the MR open event, and expert patch review at the fix MR open event. This produced a cleaner architecture than a single flow trying to orchestrate everything.

Qwen3-235B + DeepSeek v3.2 via AWS Bedrock Mantle

All six pipeline agents run through AWS Bedrock via Mantle in CI. The model tier is matched to what each agent actually needs:

  • Qwen3-235B (Agents 1, 2, 4, 5): Complex multi-step reasoning — tracing vendor criticality through source code, generating structurally correct FastAPI mock servers, planning chaos attack sequences, making mid-attack decisions as evidence accumulates across 30+ scenarios, citing specific function names and line numbers in root-cause analysis.
  • DeepSeek v3.2 (Agents 3, 6): Architecturally separate from Qwen3. Agent 3 uses DeepSeek deliberately — a different model reviewing Qwen3's generated code eliminates same-model blind spots in the security gate. Agent 6 uses DeepSeek because it consistently returns valid structured JSON at 3,000+ token output lengths.
  • Nemotron Super: Activates automatically as fallback when Qwen3 hits a transient error on Agent 2 — same interface, one-line model name swap.

Nebius Contree Sandbox

Agent 2 generates Python code (Evil Twin FastAPI servers) that will run on the CI runner. Before it does, Agent 3 sends it to Nebius Contree — a hosted isolated execution environment — and observes its actual runtime behavior. Network exfiltration, subprocess spawning, sensitive file access: caught in the cloud, not on the runner. This is the only part of the pipeline not reproducible with any commodity cloud service.

Tavily — Real-World Vendor Intelligence

Two targeted searches per vendor — {vendor} API failures incidents reliability SLA downtime and {vendor} API timeout rate limit errors 503 502 common failures — injected into the Agent 1 prompt before criticality scoring. The LLM uses this real-world outage and SLA data to calibrate the criticality score that drives the attack order and the overall resilience score weighting. Without Tavily, criticality is pure business-logic guesswork.

GitLab CI Trigger

The demo app's .gitlab-ci.yml fires a downstream pipeline on the ghostvendor project on merge_request.opened. The pipeline immediately posts an MR comment with a link to the live dashboard — the developer can watch agents activate in real time from the MR thread. The same pipeline job runs identically whether triggered by MR open or manually from the dashboard.

Dashboard

Streamlit + Supabase (shared Postgres) + GitLab Notes API + D3.js, deployed on Render at ghostvendor.onrender.com.

  • Live Run tab: D3.js agent graph polls Supabase every 4 seconds and lights up each hex node as the pipeline progresses. The CI runner writes state transitions to Supabase in real time.
  • Duo Insights tab: Fetches Flow 1 and Flow 2 bot comments from the GitLab Notes API per run — threat analysis and patch review side by side.
  • Run History tab: All completed runs with resilience score before/after, MR links, timestamps.

The dashboard is not a separate service — it is the monitoring layer for the pipeline, and the pipeline treats it as such (auto-posting the link at run start so the developer never has to find it manually).

Resilience Scoring

vendor_score = baseline_pts (20) + scenario_score (0–80), capped at 100. Each of the five chaos modes contributes up to 16 points — a vendor that handles all five scores 100, one that handles none scores 20. overall_score is a criticality-weighted average across all vendors: higher-criticality vendors (Stripe over SendGrid, for example) move the needle more, because their failure modes carry more real-world risk. The rescore pass after patching runs only the originally-failed scenarios — a vendor that failed 2 scenarios and fixed both scores 100, not a fraction of the total. If the overall score is already above threshold before patching, GhostVendor exits cleanly with no MR opened and no noise.

Orchestration

A Python state machine controls every transition: DISCOVER → ATTACK → GUARD → VERIFY → DIAGNOSE → REMEDIATE → VALIDATE. The LLM never decides what runs next. Every state is logged. Retries happen at the failed stage, not from scratch. If patching fails after 3 retry cycles, GhostVendor opens a draft findings MR with the full diagnostic report. No run ends silently.

Challenges we ran into

Timeout scenarios corrupting subsequent test runs. When the Evil Twin injects a 30-second timeout, the demo app hangs with a half-open connection. The next vendor's test inherits the stuck thread. We solved this by resetting the Evil Twin process after every timeout scenario and launching a fresh app instance per vendor.

LLM-generated code safety. Agent 2 generates a full FastAPI server that runs on the CI runner. A hallucinated twin could exfiltrate env vars or open a reverse shell. We built a 3-layer gate (AST → Contree → LLM policy) where any HIGH or BLOCKED risk discards the twin before it touches the runner filesystem.

Duo scope constraints forcing better architecture. The ai_workflows OAuth scope does not allow CI pipeline triggers — only reads and specific write operations (comments, MR reviews). Our first design tried to trigger the pipeline from a Duo flow. It doesn't work. The constraint forced us to split into two event-driven flows, which turned out to be a cleaner design: each flow does exactly one thing well, triggered by the right event, with no cross-flow state dependencies.

Patch validation with stale bytecode. Python caches .pyc files. Applying a patch to a temp copy of the repo and re-running the app would sometimes execute the old unpatched bytecode. We added an explicit __pycache__ purge step and PYTHONPATH remapping to force the patched source.

PYTHONPATH remapping across arbitrary repo layouts. Repos use src/, lib/, ., and other layouts. We made the validation step repo-agnostic by detecting the layout from editable install .pth files, then disabling those and injecting the correct temp path.

JSON truncation at large output sizes. Agent 6 returns the full patched source file — sometimes 3,000+ tokens of output. DeepSeek occasionally truncates at that scale. We added a JSON repair pass (auto-closing brackets, escaping literal newlines) and a zero-diff detector (patches identical to the original are rejected, forcing a real retry).

Port TIME_WAIT on rapid restarts. Starting and stopping Evil Twin processes quickly leaves ports in TIME_WAIT. We added OS ephemeral port fallback so a twin that can't bind its preferred port picks a free one automatically.

Accomplishments that we're proud of

7 of 9 DevSecOps lifecycle stages covered, hands-off within the MR. From MR open to validated fix MR, no human approves any individual step. The developer reviews the outcome — the fix MR — not the pipeline.

Two Duo flows doing genuine agentic work at both ends. Flow 1 delivers threat analysis the developer sees in the MR thread within 30 seconds. Flow 2 submits a formal code review on the fix MR — an AI audit trail before any human merges. Neither flow is a trigger wrapper. Both do substantive work using GitLab-native API calls.

The pipeline actually works end-to-end. Open an MR on the demo app, watch GhostVendor run through all seven pipeline states, and receive a validated fix MR — with the patch proven against chaos before it's committed. That's the full loop working autonomously.

The security gate is real. Contree sandbox execution of LLM-generated code before it runs in CI is not a checkbox — it's a meaningful guarantee that a hallucinated or malicious Evil Twin can't reach the runner environment.

Evals on an unseen repo. The eval suite runs against Twilio + Mailgun — vendors the pipeline was never built against — and passes all 4 behavioral assertions. Agent 1's AST scan + Tavily enrichment generalizes.

Model tier matching is deliberate. We didn't use the biggest model for everything. Qwen3-235B for reasoning-heavy agents, DeepSeek v3.2 for the security gate and patch generation (architecturally separate from the twin generator), Nemotron Super as automatic fallback. Each choice has a measured reason.

Limitations

  • Python only. The AST scanner, Evil Twin generator, and patch agent are built for Python Flask and FastAPI. Node.js, Go, and non-HTTP vendors are not supported yet.
  • Env-var-configurable endpoints only. GhostVendor overrides vendor base URLs via environment variables. Apps with hardcoded URLs are not detected or patched.
  • HTTP-level chaos only. Evil Twins simulate timeouts, 502s, 429s, malformed JSON, and empty responses. Database connection drops, auth token expiry, partial payload corruption, and DNS failures are out of scope.
  • One function patched per vendor. Agent 6 targets the single most critical function identified by the root-cause agent. Complex vendors with failures spread across multiple call sites may need manual follow-up.
  • Rescore is not a full regression. The after-patch score reruns only the originally-failed scenarios — it does not verify that previously-passing scenarios still pass.
  • Evals cover one target repo. The behavioral eval suite runs against a single intentionally-fragile demo app. Generalization to arbitrary real-world repos is validated only by manual test runs.

What we learned

Deterministic orchestration beats autonomous agents for a pipeline like this. We considered letting an LLM route between stages. We're glad we didn't. A Python state machine that logs every transition, retries at the exact failed stage, and always falls back to a findings MR is far more debuggable and trustworthy than an agent that might decide to skip the security gate.

Designing around platform constraints produces better architecture. The Duo ai_workflows scope constraint felt like a limitation at first. In practice, forcing two separate event-driven flows — each triggered by the right GitLab event, each doing exactly one thing — produced cleaner separation of concerns than a single flow trying to orchestrate everything. The constraint was a design input, not an obstacle.

Sandbox execution of LLM-generated code is non-negotiable. AST analysis catches syntax-level risks; only sandbox execution catches runtime behavior. We didn't appreciate how important Contree was until we started testing edge cases in Evil Twin generation. LLMs hallucinate code that looks correct but behaves unexpectedly at runtime.

Real-world web data changes the quality of criticality scoring. Tavily's two-query-per-vendor approach produces materially different criticality rankings than AST analysis alone. Vendors with documented outage histories rank higher, which means the pipeline attacks the most dangerous vendor first.

Validation before MR is the right bar. Proposing a patch that might fix the problem is not the same as proving it does. Building the local validation loop — isolated copy, fresh twins, re-run only failed scenarios — was the hardest engineering in the project and the most important.

What's next for GhostVendor

Stage 2 — Broader language and vendor support

  • Node.js / TypeScript: AST scanner and startup detector for Express and Fastify apps
  • Non-HTTP vendors: chaos modes for Redis timeouts, SQS delivery failures, database connection drops
  • Hardcoded URL detection: find vendors not using env vars via string literal analysis
  • Multi-endpoint vendors: attack each endpoint independently

Stage 3 — Production deployment

  • GitLab App with webhooks on merge_request.opened — filters to fire only when vendor client files change
  • Scalable CI runner fleet behind a job queue
  • DynamoDB + S3 for full run history at scale: score before/after, patches, MR links, token cost, duration
  • Per-repo config: vendor criticality overrides, excluded paths, minimum score threshold

Stage 4 — Multi-tenant and observability

  • Score trajectory dashboard with MR links, time-to-fix, and token cost exportable for compliance
  • On-call alerting when rescore drops below threshold after a dependency upgrade
  • Feedback loop: merged patches promote to golden examples for Agent 6 prompt improvement
  • Duo Platform expansion: Flow 3 for post-merge production monitoring — Duo watches for vendor-related errors in production logs and opens a new GhostVendor run automatically when a known failure pattern is detected

Built With

  • awsbedrock
  • deepseek
  • fastapi
  • gitlabci/cd
  • gitlabduoagentplatform
  • nebiuscontree
  • python
  • qwen3235b
  • render
  • streamlit
  • supabase
  • tavily
Share this project:

Updates

Submission history