CI Doctor

• 

  The original GitHub Actions failure exposes the real assertion and source location CI Doctor investigates.

• 

  GitHub Actions shows real bot-authored repair branches and passing checks after validated pull requests.

• 

  model diagnosis informs the workflow, but cannot mark an incident repaired or publish code.

• 

  Protected paths, repeated patches, and budget overruns stop the repair loop before any PR can be created.

• 

  Safe, evidence-linked diagnosis replaces noisy logs without exposing raw secrets or hidden reasoning.

• 

  Every state change is recorded: failure, ingestion, diagnosis, validation, and broker-created pull request.

• 

  A recorded real incident moves through GitHub evidence, policy-scoped Codex, sealed Docker, then the PR broker.

• 

  Five inspectable stages make authority boundaries and stop conditions visible to operators and reviewers.

• 

  CI Doctor turns real CI failures into evidence-backed, sandbox-validated pull requests.

• 

  Codex can propose; Docker validates; only the separate broker can publish a reviewable PR.

• 

  Codex followed checked-in rules: no mock results, no sandbox GitHub access, and recorded test proof before repair acceptance.

• 

  GitHub workflow deliveries are HMAC-SHA256 verified with a timing-safe comparison before CI Doctor accepts an incident.

• 

  Only the idempotent GitHub broker can publish validated changes; the model and sandbox never receive repository write authority.

• 

  A real signed failure produced recorded diagnosis and Docker validation before ci-doctor-demo[bot] opened PR #4 for review.

Inspiration

A broken CI pipeline is one of the most expensive small problems in software. Developers must translate noisy failure logs into a root cause, write a safe repair, add a regression test, and prove the wider suite still works.

AI can accelerate that work, but handing an agent unrestricted repository access is not acceptable for real teams. CI Doctor was inspired by that tension: how can an AI repair a real CI failure without becoming a new production security risk?

What it does

CI Doctor turns a real failed GitHub Actions workflow into an evidence-backed, sandbox-validated pull request.

A signed GitHub workflow_run failure enters CI Doctor. The system validates the webhook HMAC, checks the repository allowlist, records an append-only incident trail, bounds and pattern-redacts the relevant evidence, then asks GPT-5.6 and Codex to investigate and propose a repair.

The proposal is not trusted just because a model produced it. A checked-in repair policy restricts commands, writable paths, protected files, file count, changed lines, wall-clock time, and retry attempts. The patch must include or strengthen a regression test.

The patch is then tested in a disposable Docker workspace with network access disabled and no GitHub write credential. It must pass the focused regression test and the full suite with recorded exit codes. Only after that proof exists can a separate idempotent GitHub App broker open a PR for human review. CI Doctor never auto-merges.

How we built it

The system is built as an event-driven TypeScript application:

GitHub Actions failure
  -> HMAC verification + repository allowlist
  -> append-only evidence record + pattern redaction
  -> GPT-5.6 structured diagnosis
  -> Codex repair and regression-test proposal
  -> policy gate
  -> network-sealed Docker validation
  -> separate GitHub PR broker

The API uses Fastify and Zod schema validation. PostgreSQL stores correlated incidents, events, artifacts, repair attempts, and broker side effects. Docker is the execution authority: model output never marks a repair successful on its own.

The included fixture repository contains realistic failures across null handling, pagination boundaries, async ordering, and type assumptions. CI Doctor produced a real validated repair and a bot-authored pull request: PR #4.

The public proof dashboard shows a recorded verified incident for judges without exposing credentials or pretending that an offline worker is live.

How Codex and GPT-5.6 accelerated the project
GPT-5.6 was used for evidence-linked diagnosis: converting bounded, sanitized CI evidence into a structured hypothesis tied to source and test context.

Codex was used as the repair agent: producing a constrained patch and regression-test proposal that had to survive policy checks and Docker validation. Codex also accelerated the development workflow by helping build strict TypeScript boundaries, schema validation, focused tests, dashboard interaction states, and the repository AGENTS.md conventions.

The key human architectural decisions were intentionally outside the model: broker-only GitHub writes, no secrets in the dashboard, explicit repair budgets, append-only evidence, and Docker as the final validation authority.

Challenges we faced
The hardest challenge was avoiding the “AI wrapper” trap. A nice diagnosis screen is easy; proving that a generated repair is safe enough to review is harder.

We addressed that by separating proposal, execution, and publication authority:

The model can propose, but cannot write to GitHub.

The sandbox can execute tests, but has no network or GitHub credentials.

The broker can open a PR, but only after receiving recorded validation proof.

A human still decides whether to merge.

We also learned to be precise about security claims. GitHub webhook verification is HMAC-SHA256 with a shared secret, not public-key verification. Log redaction is pattern-based defense in depth, not a claim that every possible secret format is solved forever.

What we learned
The important lesson is that trustworthy AI developer tools need more than a powerful model. They need explicit authority boundaries and evidence that survives model failure.

CI Doctor's core principle is simple:
AI can propose. Evidence decides. Humans approve.

Built With

• cloudflare-tunnel
• codex
• docker
• fastify
• framermotion
• github
• github-actions
• github-apps
• gsap
• node.js
• openai-gpt5.6
• postgresql
• react
• tailwindcss
• three.js
• typescript
• vite
• zod