Stack Overlord: Pipeline monitoring for busy developers

Know when the code REALLY shipped—and what to do when it did not. Collect and distinguish facts from AI analysis with confidence.

Comment

Inspiration

A merged pull request feels finished, but the deployment that follows can still fail after the developer has moved on. Agentic development makes that gap easier to miss: work moves faster across multiple agents, repositories, and cloud services, while the final delivery state remains scattered across GitHub tabs and deployment platforms. But agentic development also gives engineers greater visibility to their whole pipeline.

Stack Overlord gives developers one trustworthy answer to two questions: Did it ship? If not, what should I do next?

What it does

Stack Overlord is a commit-to-deployment command center for GitHub Actions:

  • It verifies signed GitHub webhooks, deduplicates deliveries, and records workflow state in a durable Postgres database.
  • GitHub's workflow conclusion sets the factual status. GPT-5.6 explains verified failures but can never change that status.
  • For a failure, Stack Overlord collects available GitHub job and failed-step evidence and requests a structured GPT-5.6 diagnosis with evidence, confidence, limitations, and prioritized recovery actions.
  • The responsive dashboard filters runs by repository and status, links every incident back to its GitHub workflow, and keeps the model's explanation visually separate from the failure facts.
  • A Slack alert can return the developer to the incident after telemetry has already been safely stored.

The hosted demo shows real failures from an isolated public sandbox repository and recorded GPT-5.6 diagnoses. Its Replay sandbox failure control also adds a deterministic, browser-only incident so judges have a reliable test path without credentials.

What makes it different

CI dashboards report that a job failed, while general-purpose AI assistants can suggest fixes after a developer supplies the context. (We all are getting tired of copy/paste.) Stack Overlord closes that gap without asking AI to decide what happened: the signed GitHub event is stored first as pipeline truth, and GPT-5.6 only interprets the available evidence. The result is a durable delivery ledger plus a focused recovery move—with a confidence indicator so busy developers know when to take a 2nd look.

How we built it

Stack Overlord is a Next.js, React, and TypeScript application deployed on Vercel. Drizzle ORM stores raw webhook deliveries and normalized runs in Neon Postgres. The webhook route verifies GitHub signatures before parsing, persists accepted telemetry before optional work, and then enriches verified failures with GitHub Actions evidence. GPT-5.6 is called through the OpenAI Responses API with a strict structured output schema. Slack notification, database, GitHub, and OpenAI clients are initialized lazily so missing optional credentials do not break the build or the deterministic demo.

Codex accelerated the core build: it translated the product boundary into the architecture, schema, signed-webhook contract, evidence enrichment, GPT-5.6 output schema, Slack integration, responsive repository dashboard, deterministic fixtures, and regression tests. It deployed agents (sometimes multiple simultaneously) to plan and execute work, responding to my steering input. It also drove the lint, type-check, test, production-build, and desktop/mobile browser-verification loop. Key decisions—GitHub owns truth, persistence happens first, and AI remains an optional interpretation layer—are reflected in both the code and the interface. I particularly valued the ability to move local work into a cloud agent thread where I could continue steering the project via my phone.

Challenges we ran into

  • Preserving a verified failure even when GitHub enrichment, OpenAI, or Slack is unavailable.
  • Giving GPT-5.6 enough evidence to help without letting a plausible explanation become invented pipeline truth.
  • Keeping the work of multiple agents straight when they were working in the same code--Codex was very strong at resolving GitHub merge conflicts and identifying when code was mistakenly included in a merge.
  • Providing a real end-to-end demo without targeting the original Cognitive Bridge production repository or Firebase project.
  • Keeping repository, workflow, commit, timing, and recovery detail readable on desktop and mobile.

Accomplishments that we are proud of

  • A working path from signed GitHub event and Vercel deployment result to Postgres persistence, GPT-5.6 diagnosis, Slack notification, and responsive UI.
  • Real sandbox failures and traceable GPT-5.6 response IDs in the hosted deployment.
  • A no-login replay path that judges can test without rebuilding or configuring the project.
  • Getting the "S" logo almost perfect, after early misunderstandings.

What we learned

The most valuable AI boundary was also the simplest: deterministic systems should state what happened; the model should explain what the evidence might mean. GPT-5.6 becomes more useful and more credible when confidence, limitations, and verification steps are part of the contract rather than optional prose.

What is next

  • Correlate pull requests, merge commits, deployments, and live endpoint health in one timeline.
  • Add deeper log and artifact retrieval for higher-confidence diagnoses.
  • Add repository-specific runbooks, notification policies, and incident history.
  • Offer explicitly approved remediation workflows while keeping factual state outside the model's control.

Built With

  • codex
  • drizzle-orm
  • github-actions
  • github-webhooks
  • gpt-5.6
  • neon-postgres
  • next.js
  • openai-responses-api
  • react
  • shadcn/ui
  • slack
  • tailwind-css
  • typescript
  • vercel
Share this project:

Updates