Inspiration
We have all seen it happen:> a massive Black Friday sale or a viral ticket drop goes live, and the website instantly crashes. Traditional auto-scalers (like Kubernetes HPA) are purely reactive they wait for CPU usage to hit 80% before adding servers. By then, the database is usually overwhelmed, and the system collapses under a "Thundering Herd."
Furthermore, when a crash happens at 3 AM, Site Reliability Engineers (SREs) have to wake up, dig through dashboards, and manually piece together the timeline. We asked ourselves: What if we built a "Self-Driving" Cloud Autopilot that uses predictive physics to prevent crashes, and AI to automatically write the incident report?
What it does
Autonomous SRE is an aerospace-grade control system that protects distributed systems (Kubernetes/Envoy) from catastrophic failures.
The Traffic Bouncer (Predictive Scaling):> Instead of blindly adding servers, it calculates exactly how much traffic the backend can physically handle. If it detects a database lockup, it temporarily queues traffic at the edge to keep the system alive, enforcing strict Service Level Agreements (SLAs).
The Byzantine Shield (Flying on Instruments):> If your monitoring tools break and send garbage data (like NaN or negative traffic), the autopilot ignores the fake data and safely "flies on instruments" using its own mathematical predictions until the sensors heal.
Automated AI Post-Mortems:> The moment an incident is resolved, our tool automatically fetches the "black box" flight data from Splunk Enterprise and sends it to DeepSeek-V3 (via GitHub Models). The AI instantly writes a clear, human-readable Root Cause Analysis (RCA).
How we built it
The core engine is written in pure Go for extreme performance. The Math Engine :> We built a Model Predictive Controller (MPC) and a 9-Dimensional Extended Kalman Filter (EKF). To prevent database collapse, we programmed the AI to obey Neil Gunther's Universal Scalability Law (USL):$$C(N) = \frac{N}{1 + \alpha(N-1) + \beta N(N-1)}$$
Splunk Observability:> We integrated natively with the Splunk REST API using oneshot execution mode. The engine outputs telemetry (Risk Score, Latency, Queues, Retry Pools) which is immediately indexed by Splunk.
AI Diagnostics:> We used the GitHub Models API to pass the localized Splunk search results to DeepSeek-V3. We engineered a strict prompt that forces the LLM to act as a Senior SRE, outputting Severity, Root Cause, Impact, and Engineering Fixes.
Challenges we ran into
The "Euler-RED Singularity":> We discovered a massive integration bug where our simulated Envoy proxy dropped its physical queue below the queue limit in a single 1.0s time-step. This caused the proxy to instantly admit 26,000 requests from the retry pool. We fixed this by engineering an SLA-aware Little's Law boundary ($L = \lambda W$) that enforces a strict structural floor.
Global Mutex Contention:> During our Monte Carlo simulations (calculating the 95% Value-at-Risk), Go's math/rand library bottlenecked our CPU across 16 cores because of a global Mutex lock. We bypassed this by building a zero-allocation Common Random Numbers (CRN) matrix cache, dropping the SDE integration time from 30+ seconds to under 3 seconds.
Accomplishments that we're proud of
Zero-Allocation Hot Path:> We successfully engineered the MPC and EKF to evaluate over 750 Million Stochastic Differential Equations across multithreaded CPU cores with zero heap memory allocations during runtime.
Flawless AI Integration:>Bridging complex control-theory telemetry with an LLM and getting a deterministic, perfectly formatted RCA report via Splunk data feels like magic every time it runs.
What we learned
We learned that the hardest part of control systems isn't the math it's handling the "friction" of reality. We gained deep insights into Go memory profiling, mitigating cache-line bouncing, and securely authenticating with enterprise platforms like Splunk via REST APIs.
What's next for Autonomous SRE: Cloud Autopilot
The mathematical blueprint is complete. The next step is translating the simulator into a true Kubernetes Operator (CRD). We plan to replace our telemetry simulator with live eBPF metrics and wire the Actuator output directly into the Kubernetes client-go API and Istio DestinationRules.
Log in or sign up for Devpost to join the conversation.