CloudTxn

Human or AI, every incident PR must prove itself

Comment

Inspiration

During an incident, a green pull request does not mean the customer problem is fixed. Changes across Kubernetes, configuration, and feature flags may all succeed technically while the real failure remains elsewhere. CloudTxn answers a harder question: did the pull request restore the customer experience, and can every change be safely reversed?

What it does

CloudTxn is a GitOps safety gate for incident-response pull requests. It reads a real GitHub PR, accepts only an allowlisted declarative incident manifest, and compiles it into a typed, reversible transaction. Before applying changes, CloudTxn captures snapshots across Kubernetes, AWS-compatible SSM, and feature flags. It applies the proposed fix in an isolated environment and verifies real customer-facing checkout through Kong. If the business-health check fails, CloudTxn automatically compensates every mutation in reverse order. In the demo, the engineer scales the payment API, switches the provider, and enables fallback. All changes succeed technically, but checkout remains HTTP 503 because a PostgreSQL AccessExclusiveLock is still active. CloudTxn rejects the PR and restores the original state. A local Ollama model analyzes the PostgreSQL evidence and identifies payment-reconciler as the blocker. A second PR pauses the reconciler. CloudTxn tests the remediation, verifies HTTP 200, restores the test snapshot, and only then marks the PR safe to merge. AI proposes the fix. CloudTxn proves it is safe.

How we built it

CloudTxn is implemented in Python with FastAPI and strict Pydantic models. These models form a safety boundary: neither the pull request nor the AI can select credentials, arbitrary commands, endpoints, or rollback logic. The demo uses real GitHub pull requests and GitHub Actions, a local k3d Kubernetes cluster, PostgreSQL, Kong, LocalStack SSM, a feature-flag API, and Ollama with Qwen2.5-Coder. The AI analyzes evidence and proposes constrained remediation steps, but never executes infrastructure operations. The transaction engine follows a Saga-style workflow: snapshot → apply → verify → commit On failure: compensate in reverse order → verify restoration Every decision, snapshot, verification, and compensation step is recorded in an append-only JSONL evidence journal.

Challenges we ran into

The hardest problem was defining “safe” across systems without a shared transaction protocol. Kubernetes, SSM, feature flags, PostgreSQL, and the customer-facing API all have different state models and failure modes. We had to capture enough state before each mutation to support deterministic rollback. We also discovered that new payment API pods could start at the container level while remaining unavailable because their initialization query was blocked by the same PostgreSQL lock. This led us to separate technical application from business verification. Kubernetes rollout status alone was not enough, so CloudTxn uses the actual customer outcome as the final decision.

Accomplishments that we're proud of

We built a complete incident workflow around two real GitHub pull requests and a real PostgreSQL failure. The first PR passes normal CI but fails CloudTxn's customer-health verification. Its changes across Kubernetes, SSM, and feature flags are automatically restored. The second PR is based on evidence collected from PostgreSQL. CloudTxn applies it in a reversible test, observes HTTP 200, restores the original incident state, and then allows the change to be promoted separately. We are especially proud that even a successful safe test is rolled back. Validation never silently becomes deployment.

What we learned

A green CI check proves that code passed predefined tests. It does not prove that an operational change resolves a live customer problem. Business outcome must be part of the deployment contract, not an afterthought. Rollback must also be designed before execution, because capturing state after a failure is too late. Finally, constrained AI was more useful than privileged AI. The model interpreted real evidence and proposed a remediation, while deterministic code handled policy, execution, verification, rollback, and evidence.

What’s next for CloudTxn

We plan to publish CloudTxn validation as a native GitHub Actions check with a pull-request comment containing the transaction evidence and JSONL journal artifact. Next steps include merge-queue protection, additional adapters for Argo CD, Flux, Terraform, cloud configuration, and observability systems, as well as disposable preview environments for each PR. Longer term, CloudTxn can become a safety layer between AI-generated operational changes and production infrastructure. AI can propose increasingly sophisticated fixes, while deterministic policy, execution, verification, and rollback remain outside the model.

Built With

Share this project:

Updates

Submission history