-
-
docker compose
-
GPT Integration
-
Architecture
Inspiration
Site Reliability Engineers are expected to make high-impact decisions under pressure by correlating alerts, traces, runtime health, logs, and recent configuration changes. While large language models are excellent at explaining complex information, I wanted to explore how they could be safely integrated into production reliability workflows without becoming the authority for operational decisions.
ReliOps AI was inspired by a simple question: Can AI help engineers investigate incidents without hallucinating root causes or taking unsafe actions?
Instead of allowing the language model to decide what happened, I designed an evidence-first architecture where deterministic investigation establishes the facts and GPT-5.6 explains validated findings, proposes safe remediation, and generates incident documentation.
What it does
ReliOps AI is an evidence-driven "AI Site Reliability Engineer" for incident investigation.
The demo simulates a checkout latency regression caused by increasing PAYMENT_DELAY_MS from 100 ms to 3000 ms. The system correlates modeled incident evidence—including configuration changes, runtime health, alerts, and traces—to build an incident timeline and evaluate competing hypotheses.
A deterministic investigation engine identifies the most likely root cause with evidence-backed confidence. GPT-5.6 then generates an executive summary, preventive actions, and an incident postmortem while preserving strict deterministic guardrails.
How we built it
ReliOps AI was built with Python 3.12, FastAPI, Pydantic v2, OpenTelemetry, structured JSON logging, the OpenAI Responses API, and GPT-5.6.
The project follows an evidence-first architecture:
A deterministic investigation engine constructs the timeline, evaluates hypotheses, determines the root cause, assigns confidence, and defines remediation. GPT-5.6 receives structured investigation results and generates evidence-backed explanations and preventive recommendations. Validation guardrails ensure GPT cannot modify deterministic conclusions, confidence values, remediation proposals, or introduce unsupported evidence citations. A lightweight dashboard demonstrates the complete investigation workflow, from evidence collection through postmortem generation.
Throughout development, I used Codex as my primary coding collaborator for repository implementation, testing, debugging, documentation, and UI refinement. I remained responsible for the overall architecture, safety model, acceptance criteria, validation rules, and final technical decisions.
Challenges we ran into
The biggest challenge was designing a system where AI improves incident response without becoming the source of truth. Another challenge was debugging the live OpenAI integration. During development, an API quota issue initially appeared as a generic service failure. By isolating the Responses API outside the application, I identified the underlying quota problem, corrected it, and completed the live integration.
Accomplishments that we're proud of
I'm proud that ReliOps AI demonstrates a practical approach to trustworthy AI for Site Reliability Engineering rather than simply wrapping an LLM around operational data.
Highlights include: Evidence-first deterministic investigation Competing hypothesis evaluation GPT-5.6 explanations constrained by validation guardrails Human-approved remediation proposals Recovery verification safeguards Evidence-backed incident postmortem generation An end-to-end interactive dashboard demonstrating the complete workflow
Most importantly, the project maintains a clear separation between validated operational facts and AI-generated explanations.
What we learned
This project reinforced that AI is most valuable when combined with deterministic engineering systems rather than replacing them.
I learned how to build structured workflows around the OpenAI Responses API, design validation layers that constrain model behavior, and use Codex effectively for large-scale implementation while maintaining architectural control.
What's next for ReliOps AI
Future work includes integrating real production evidence from platforms such as Datadog, CloudWatch, and Kubernetes, expanding provider support through MCP connectors, generating GitHub remediation pull requests, verifying deployments with Argo CD, publishing postmortems to Confluence, and integrating approval workflows with Slack.
The core design philosophy will remain unchanged: evidence first, deterministic validation, human approval for production changes, and AI used to explain and accelerate—not replace—engineering judgment.
Log in or sign up for Devpost to join the conversation.