Bulwark

• 

  Bulwark readme

• 

  GCP logs

Inspiration

Pre-production testing suffers from critical structural gaps. Engineering teams routinely merge code that passes synthetic unit tests, only to watch production break under live traffic patterns because staging environments rarely receive real traffic.

According to industry data, the median recovery time for a production incident spans 1 to 4 hours. Fixing defects after deployment costs up to 15 times more than catching them during code review. Furthermore, as LLM components proliferate, prompt regressions rarely trigger hard infrastructure alerts. Arize platform data shows that prompt template changes are responsible for quality regressions in approximately 40% of LLM production incidents, with a median detection time of 3.2 days because no automated pre-merge gate exists.

The goal was to build a single, automated pre-merge gate in the continuous integration pipeline that simultaneously validates both infrastructure performance and LLM behavioral quality before a single line of code goes live.

What it does

Bulwark is a pre-deployment production simulation agent that gates GitLab Merge Requests (MRs) based on a unified confidence score.

• Environment Provisioning: It automatically packages the branch archive, runs a Cloud Build, and provisions an isolated instance of the service on Google Cloud Run.
• Dependency Mocking: It maps downstream topology using Dynatrace and generates static WireMock service stubs to simulate stateless external HTTP dependencies.
• Traffic Replay: It pulls a sample of 50 to 200 recent production requests, scrubs them of personally identifiable information (PII), and replays them against the ephemeral instance.
• LLM Evaluation: For applications with generative AI components, it extracts live execution traces via a /traces endpoint and submits them to Arize evaluators to check for hallucination, relevance, and groundedness deltas.
• Unified Gate: It feeds the performance deltas, error classifications, and LLM quality metrics into a Gemini 2.0 Flash reasoning step. It then posts a verdict directly to the MR and sets the pipeline gate to APPROVED, WARNING, or BLOCKED.

How we built it

The system architecture relies on decoupled, asynchronous services and native tool integrations.

• Control Plane: A FastAPI webhook receiver handles GitLab payloads and forwards events through Google Cloud Pub/Sub to an orchestrator worker running on Cloud Run.
• Integration Layer: We built a shared client adapter utilizing the official Python Model Context Protocol (MCP) SDK. The orchestrator uses MCP to natively call tools across GitLab, Dynatrace, and Arize without relying on fragile custom REST wrappers.
• Build Pipeline: The system generates source archives from the target branch directly to a Google Cloud Storage bucket, utilizing Cloud Build to deploy the container alongside a real WireMock instance.
• Data Strategy: A MongoDB Atlas M0 cluster acts as the state ledger. It enforces 90-day record retention and queries historical simulation context to inject into the Gemini prompt.

Challenges we ran into

The Statistical Noise Floor

Calculating infrastructure metrics over small traffic samples is mathematically tricky. A p95 latency metric derived from a sample size of 200 requests inherently carries a statistical uncertainty band of roughly $\pm35\text{ ms}$. If an infrastructure gate applies rigid thresholds smaller than the measurement noise, every verdict becomes a coin flip. To overcome this, we built a deterministic pre-processing scoring step. Deductions are only levied against the final score if the observed regression safely breaks past this statistical noise floor:

$$\text{Confidence Interval} \approx \bar{x} \pm z \left( \frac{s}{\sqrt{n}} \right)$$

Eliminating Stateful Mocking Illusions

Early planning assumed the system could build fully stateful, context-aware mocks. In practice, recreating database transactional integrity or token rotation purely out of static traces is impossible. We addressed this by constraining dependency mocking entirely to stateless HTTP schemas and adding a mandatory uncertainty disclosure block to every MR comment, explicitly warning developers about what the simulation cannot test.

LLM Component Detection False Positives

We initially planned to use simple string matching to detect LLM code, but realized common terms like "system_prompt" would trigger massive false positive rates in standard codebases. We solved this by implementing a two-stage detection pipeline: a fast file path heuristic followed by an Abstract Syntax Tree (AST) import scan capped at 50 files to verify true AI SDK usage.

Accomplishments that we're proud of

• True Protocol-Driven Design: The entire platform communicates across vendor boundaries using native MCP connections, establishing clean separation between infrastructure tools, evaluation engines, and code repositories.
• Proportional Error Classification: When novel errors appear, WireMock stubs often produce artifacts (like 404s for unmocked endpoints). The system relies on Gemini 2.0 Flash to evaluate code diffs alongside novel error strings to accurately classify them as either stub artifacts or authentic code regressions.
• Honest Limitations: The system does not pretend to be a silver bullet. It explicitly injects LLM evaluator noise bands (e.g., $\pm0.10$) into its reasoning and publicly admits measurement uncertainty directly in the GitHub/GitLab pipeline comment.

What we learned

We learned that building long-term memory structures directly into language models is the wrong design choice for deterministic developer tooling. Gemini 2.0 Flash is a stateless language model; it does not "learn" from past simulation failures.

Shifting to a completely stateless model architecture—where historical regression trends and previous execution records are explicitly queried out of MongoDB and injected cleanly into a single structured prompt window—yielded far more stable and reproducible verdicts. We also learned that clearly communicating what a testing tool cannot catch (like conditional regressions or concurrency bugs) is just as critical for establishing developer trust as demonstrating what it can catch.

What's next for Bulwark

Built With

• arize
• dynatrace
• fastapi
• gcp
• gitlab
• mcp
• python
• typescript