Chronos-SRE

CHRONOS-SRE autonomously diagnoses production outages, generates and sandbox-verifies code fixes, and prepares review-ready GitHub pull requests for faster, safer incident recovery.

Comment

Inspiration

Every software engineer knows the dread of a 3:17 AM Sev-1 outage alert.

PagerDuty blares, fifteen panicked stakeholders jump onto a war-room bridge, and on-call engineers are thrust into a wall of thousands of noisy OpenTelemetry logs, timeout cascades, and cryptic stack traces. Under intense time pressure and sleep deprivation, manual diagnosis breaks down. Worse, desperate engineers often turn to generic conversational AI chatbots—only to be burned by hallucinated packages, invalid function signatures, and broken assumptions that make outages even more catastrophic.

The realization was clear: Modern cloud infrastructure scales in milliseconds, but incident remediation is still manual, fragmented, and dangerously slow.

We didn't set out to build another conversational chatbot. We set out to build Chronos-SRE—an autonomous Site Reliability Engineering platform that acts as a real, rigorous reliability engineer. It investigates the crime scene, isolates the root cause using NVIDIA Nemotron-70B, rigorously validates syntax invariants to eliminate hallucinations, and deterministically proves the fix in an isolated sandbox before opening a production-ready GitHub Pull Request.

What it does

Chronos-SRE closes the entire incident remediation loop from detection to pull request without human intervention:

  1. Universal Telemetry & Issue Ingress: Connects to any public GitHub repository to auto-discover open bug reports and failing CI runs, or directly ingests raw OpenTelemetry JSON traces and production crash dumps.
  2. Telemetry Sifting: Automatically parses callstacks, isolates exception types, and extracts the offending source file and line boundaries.
  3. Dual-Agent Causal Swarm: Uses NVIDIA Nemotron-70B to formulate competing causal hypotheses and deduce the exact failure mechanism (e.g., inverted arithmetic windows, circular lock acquisitions, unhandled dictionary lookups).
  4. Adversarial Invariant Critic: A dedicated critic loop checks candidate hypotheses against the repository's Abstract Syntax Tree (AST), rejecting any hallucinations or invalid syntax before execution.
  5. Deterministic Ephemeral Sandbox: Spins up an isolated testbed runner to execute reproduction tests (EXIT 1 • Bug Confirmed), applies the patch, and re-executes tests (EXIT 0 • Fix Verified).
  6. Autonomous GitHub Pull Request: Directly opens a clean, verified Pull Request on GitHub with automated release notes and sandbox regression proofs.

How we built it

Chronos-SRE is built on a modern, decoupled cloud architecture designed for sub-second agentic reasoning:

🧠 The Core Brain: NVIDIA Nemotron-70B (nvidia/llama-3.1-nemotron-70b-instruct)

NVIDIA's Nemotron-70B is the cognitive engine powering Chronos-SRE. Standard LLMs frequently stumble when asked to trace multi-file call stacks or perform precise invariant checking. Nemotron-70B stands apart due to its exceptional reasoning depth and strict instruction adherence. In our pipeline, Nemotron operates across three distinct roles:

  • Causal Diagnosis: Deducing line-level root causes from noisy distributed traces.
  • Adversarial Critic: Cross-examining proposed hypotheses against AST boundaries with a strict 0.99 confidence invariant gate.
  • Surgical Patch Synthesis: Generating exact SEARCH / REPLACE diff blocks with defensive guard clauses rather than rewriting entire files.

🚀 High-Throughput Inference: Nebius Token Factory

Critical production incidents demand immediate answers. We served Nemotron-70B via Nebius Token Factory, achieving a sustained streaming throughput of ~91.4 tokens/second and near-zero Time-To-First-Token (TTFT). This allows our dual-agent swarm to formulate hypotheses and synthesize multi-point patches in under 2 seconds.

🔍 Grounded Context: Tavily Search

Grounds investigations with real-time web search to identify upstream dependency deprecations, security advisories, and package errata.

💻 Stack & Infrastructure

  • Backend: FastAPI (Python 3.11) with asynchronous Server-Sent Events (SSE) streaming real-time telemetry to the UI. Deployed on Render.
  • Frontend: Next.js 16 (React 19, Turbopack, Tailwind CSS) featuring a real-time SRE War-Room UI with Monaco Diff Editor. Deployed on Vercel.
  • Sandbox: Subprocess-isolated temporary test environments executing native Pytest and Vitest regression suites.

Challenges we ran into

  1. Eliminating Hallucinations in Mission-Critical SRE: In production incident remediation, a single hallucinated import or incorrect parameter can cause a cascading outage. We solved this by pairing Nemotron-70B with an AST-grounded Critic Agent. Candidate patches are validated against the repository's live syntax tree to ensure zero phantom imports or syntax regressions before any code touches the sandbox.

  2. Deterministic Sandbox Verification: Ensuring that reproduction and verification runs are 100% deterministic required careful process isolation. We built an ephemeral sandbox runner that creates fresh temporary directory clones, isolates environment variables, and mathematically enforces: $$\text{Verified} = (\text{ReturnCode}{\text{pre}} \neq 0) \land (\text{ReturnCode}{\text{post}} = 0)$$

  3. Cloud SSE Streaming & Reverse Proxy Buffering: Delivering live, token-by-token reasoning and OpenTelemetry log streams through cloud reverse proxies (Render and Vercel) often leads to event buffering. We resolved this by explicitly injecting X-Accel-Buffering: no, Cache-Control: no-cache, and Connection: keep-alive headers into our FastAPI StreamingResponse.

Accomplishments that we're proud of

  • True Autonomous Remediation: Successfully demonstrated zero-touch remediation on complex real-world defects—including an actual multi-point disaster calculation bug in GramRaksha-AI (inverted 72-hour countdown windows, inverted urgency thresholds, and unhandled null dictionary lookups).
  • Deterministic Proof, Not Guesswork: Chronos-SRE never claims an issue is fixed until Pytest/Vitest mathematically returns EXIT 0 in the sandbox.
  • Full Production Cloud Deployment: Both the Next.js frontend (chronos-sre.vercel.app) and FastAPI backend (chronos-sre-backend.onrender.com) are live, healthy, and publicly accessible.

What we learned

  • The Power of NVIDIA Nemotron-70B: Working with Nemotron-70B demonstrated that open-source frontier models have reached the reasoning threshold required for true agentic software engineering. Its ability to follow strict formatting constraints (like SEARCH/REPLACE blocks) and reason through complex logic flaws made autonomous remediation viable.
  • Execution is the Best Evaluation: LLM outputs should never be evaluated on text similarity alone. Grounding models in an executable environment with automated reproduction tests is the only dependable path forward for AI in DevOps.

What's next for Chronos-SRE

  • Kernel-Level eBPF Ingestion: Ingesting raw network socket and syscall data directly from Kubernetes cluster nodes for sub-millisecond anomaly detection.
  • Multi-Repository Microservice Tracing: Tracing breaking API contract changes across multiple interdependent repositories simultaneously.
  • Automated Canary Deployment & Rollback: Integrating with ArgoCD and Kubernetes to autonomously deploy verified hotfixes to canary pods and monitor telemetry for zero regression before 100% rollout.

Built With

  • fastapi
  • github-api
  • monaco-editor
  • nebius
  • nemotron-70b
  • next.js
  • nvidia
  • opentelemetry
  • pydantic
  • pytest
  • python
  • react
  • render
  • tailwindcss
  • tavily
  • typescript
  • vercel
  • vitest
Share this project:

Updates

Submission history