CloudOps Autopilot: Learning Loop

An AI incident-triage tool that diagnoses cloud failures, captures expert corrections, and turns them into reusable evals so the same reasoning mistake does not happen twice.

Comment

Inspiration

Cloud and CI CD failures often produce large amounts of technical data but very little usable guidance. Engineers may spend hours moving between logs, infrastructure configuration, documentation, and team messages before identifying the real cause.

I previously built an initial CloudOps Autopilot prototype that converted infrastructure failure logs into a structured diagnosis. During OpenAI Build Week, I wanted to extend that idea beyond one-time analysis.

The new goal is to create a learning loop where expert corrections do not disappear into tickets or chat threads. Instead, each correction becomes a reusable evaluation case that helps detect the same reasoning failure in future model or prompt versions.

What it does

CloudOps Autopilot: Learning Loop is a developer tool for diagnosing CI CD and cloud infrastructure failures.

A user can:

  1. Submit a Terraform, container, IAM, KMS, or deployment failure log.
  2. Receive a structured GPT-5.6 diagnosis containing:
    • Probable root cause
    • Supporting evidence
    • Eliminated hypotheses
    • Recommended next action
    • Confidence
  3. Review the diagnosis and submit an expert correction when necessary.
  4. Convert that correction into a reusable regression evaluation.
  5. Run saved evaluations and see which cases pass or fail.
  6. Identify when a future prompt or model change causes diagnostic quality to regress.

The central idea is simple: the same reasoning mistake should not happen twice.

What is new for OpenAI Build Week

The original prototype performed one-time log analysis using a different AI platform.

The Build Week version adds new functionality built with Codex and GPT-5.6:

  • GPT-5.6 powered incident diagnosis
  • Expert correction capture
  • Correction to evaluation conversion
  • Saved regression cases
  • Evaluation pass and fail reporting
  • Documentation of the development process through Codex session history and dated commits

This correction-to-evaluation workflow was created during the Build Week submission period.

How I built it

I used Codex as my primary development partner to plan, implement, test, and refine the new functionality.

Codex helped me:

  • Define the minimum viable correction and evaluation workflow
  • Scaffold the application structure
  • Implement structured GPT-5.6 responses
  • Build the expert correction form
  • Create persistent evaluation records
  • Add regression test execution and results
  • Review code and identify edge cases
  • Generate and improve automated tests
  • Refine the user experience and documentation

The application uses GPT-5.6 for incident analysis and structured reasoning. Saved expert corrections provide the expected diagnosis for future evaluation runs.

The repository includes dated commits and a README section explaining which functionality existed before Build Week and which functionality was added during the event.

Challenges

One challenge was designing an evaluation that measures diagnostic quality rather than requiring an exact text match. A correct diagnosis can be phrased in several ways, so the evaluation workflow needs to compare the important technical claims rather than only comparing strings.

Another challenge was keeping the scope focused. Cloud operations covers thousands of possible failure modes, so I concentrated on a few realistic categories:

  • Terraform state and configuration failures
  • Cloud authorization and encryption failures
  • Container and dependency failures

The final challenge was preserving human authority. CloudOps Autopilot does not automatically change infrastructure. It explains, recommends, captures corrections, and evaluates future responses while keeping the engineer in control.

What I learned

I learned that expert corrections are valuable product data. A correction is more useful when it becomes a permanent regression test instead of remaining in a ticket or message history.

I also learned that structured outputs make incident analysis easier to review, compare, and evaluate. Separating evidence, eliminated hypotheses, recommended actions, and confidence produces a more useful engineering workflow than an unrestricted chatbot response.

Codex was most valuable when the desired outcome and proof of completion were clearly defined. It accelerated implementation, testing, review, and documentation while leaving the core product and engineering decisions with me.

Potential impact

CloudOps Autopilot: Learning Loop could help platform, DevOps, SRE, and application teams:

  • Reduce time spent diagnosing recurring failures
  • Preserve expert operational knowledge
  • Improve onboarding for less experienced engineers
  • Detect regressions caused by prompt or model changes
  • Build greater trust in AI-assisted incident analysis

The long-term vision is an operational reasoning system that improves from verified human expertise while remaining transparent, testable, and controlled.

Built With

Share this project:

Updates