Rollsafe

Inspiration

In production systems—like hospital patient intake, banking gateways, and emergency dispatch—a broken release is catastrophic. Most teams still rely on "all-at-once" deployments where a bad configuration or unhandled null pointer crashes the entire user base at once.

Even when alerts fire, Mean Time to Recovery (MTTR) is usually 15 to 45 minutes: an on-call engineer wakes up to a PagerDuty alert, opens logs, hops on a call, and runs manual rollback commands. During those 20 minutes, requests fail and users are stranded.

We wanted to build something that functions like a mechanical circuit breaker in a home electrical panel. When an appliance shorts out, you don't wait for an electrician to drive over and flip a switch; the fuse trips immediately. We built RollSafe to bring that same automatic fail-safe behavior to web deployments.

What it does

RollSafe is an automated progressive canary delivery and zero-downtime resilience controller:

Traffic Splitting: When deploying a new release candidate, RollSafe routes 90% of live incoming traffic to the proven stable version (v1) and samples just 10% through the canary candidate (v2).
Continuous Health Telemetry: A controller daemon polls Prometheus error rates and HTTP response codes on both versions in real-time.
Sub-50ms Auto-Rollback: If canary errors exceed 5% across 3 evaluation cycles, the circuit breaker trips. The gateway immediately drops canary traffic to 0% and routes 100% of user requests back to stable in under 50 milliseconds.
Live SRE Station: A dark-themed telemetry dashboard displays live traffic weights, upstream health status, error counters, and an audit trail of fail-safe events.

How we built it

We implemented RollSafe with a dual-engine architecture to support both lightweight edge appliances and full-scale cloud clusters:

Core Microservice (Go 1.21): A lightweight HTTP service instrumented with Prometheus client metrics, structured logging, and built-in fault simulation endpoints.
Dynamic Routing Gateway (NGINX): Uses weighted upstream reverse-proxying with instant reload triggers for sub-second traffic diversion.
Telemetry & Auto-Trip Controller (Python): An active health daemon that samples metrics, maintains sliding-window health evaluation, and executes atomic config updates on failure.
Enterprise Kubernetes Manifests: Complete production configs using Argo Rollouts custom resource definitions (AnalysisTemplate, Rollout), Flagger progressive delivery, and Prometheus metric validation rules.
Interactive SRE Dashboard: Vanilla HTML/CSS/JavaScript with Server-Sent Events/polling for live metrics without frontend framework overhead.
Load Testing (k6): Automated multi-stage stress scripts simulating high-concurrency user traffic and sudden error injection.

Challenges we ran into

Preventing Metric Flapping: A naive rollback script trips the moment a single request fails. In real networks, occasional packet drops or brief DNS timeouts happen. Tripping immediately caused constant false alarms. We solved this by designing a consecutive-window threshold (requiring 3 consecutive failed sampling cycles) before triggering an emergency rollback.
Kubernetes Resource Weight vs. Edge Feasibility: Running a full Kubernetes cluster with Argo Rollouts and Prometheus requires 6 to 8 GB of RAM—unrealistic for small edge machines or quick evaluations. Instead of choosing one over the other, we built both: a self-contained 150MB Docker Compose / standalone runner that starts in 3 seconds, and a full Kubernetes manifest suite for production clusters.
Zero Dropped Connections During Failover: Reconfiguring proxy upstreams during high traffic can drop in-flight requests if not handled carefully. We tuned NGINX upstream weights and keepalive settings so that shifting canary traffic back to stable happens cleanly without severing existing TCP connections.

Accomplishments that we're proud of

Sub-50ms Reaction Time: Going from detecting sustained canary failure to 100% stable traffic diversion in under 50 milliseconds.
Single-Command Demo: Anyone can evaluate the entire system locally in seconds using standard Python without needing Kubernetes pre-installed.
Clean SRE Visualization: Building an intuitive dashboard that clearly shows the circuit breaker tripping and recovery in action during live fault injection.

What we learned

Designing safety mechanisms requires balancing fast reaction against noise tolerance.
How Argo Rollouts handles declarative progressive delivery steps and metric queries inside Kubernetes.
The practical engineering trade-offs between heavy cloud-native tooling and lean standalone daemons for edge resilience.

What's next for RollSafe

Adding automated progressive step-up promotions (10% $\rightarrow$ 25% $\rightarrow$ 50% $\rightarrow$ 100%) based on automated SLO metric checks.
Integrating webhook notifications for Slack, Discord, and PagerDuty with incident snapshots.
Adding header-based routing (e.g. routing internal beta testers to canary via cookie or auth claims).