PR-Herder

PR Herder — a Slack agent that connects GitHub, a local LLM, and Slack to triage pull requests end to end.
Full architecture: webhook ingest, Postgres queue, deterministic triage, Ollama summaries, Slack, and GitHub MCP execution.
Request flow: a PR triggers triage, an optional local LLM summary, and a real GitHub review via MCP — all from Slack.
The actual Slack card: size classification, AI summary for ambiguous PRs, and one-click Approve / Request changes.

Inspiration

Every open-source maintainer and busy engineering team loses real time every day tab-switching between GitHub and Slack — checking CI status, re-reading diffs to figure out who should review something, and manually pinging people about pull requests that have gone stale. We wanted to collapse that entire loop into one surface: Slack.

What it does

PR Herder is a Slack agent that watches GitHub pull requests, triages them using deterministic rules, optionally summarizes ambiguous changes with a locally-hosted LLM, and posts an interactive card straight into Slack. Reviewers approve or request changes with a single click — the action executes on GitHub in real time through the official GitHub MCP Server, using JSON-RPC over Server-Sent Events.

It also runs a daily digest of stale pull requests, respecting configured quiet hours so nobody gets pinged overnight.

PR Herder follows a deterministic-first design: rule-based triage always makes the primary decision — routing by sensitive file paths, diff size, and contributor history. The LLM is only ever invoked to summarize a PR the rules couldn't confidently classify — never to route, approve, or decide anything security-relevant. This keeps the system predictable and auditable while still benefiting from AI exactly where it adds value.

How we built it

Go backend, structured around clean package boundaries (ingest, triage, slackui, githubmcp, authz, scheduler, llm)
PostgreSQL for idempotent webhook storage and processing state
Slack Block Kit for the interactive triage cards, verified via HMAC signature checking on every interaction
GitHub MCP Server integration — a hand-built JSON-RPC 2.0 client over Server-Sent Events, since GitHub's MCP server uses the Streamable HTTP transport rather than plain JSON responses
Ollama, running a local llama3.2:3b model, for PR summarization — no external API dependency, no billing, no quota limits
Production hardening: exponential-backoff retries on every external call, a dead-letter queue for permanently failing events, Prometheus metrics, structured JSON logging with request correlation, and liveness/readiness health checks
Deployment: a multi-stage Docker build to a distroless image, plus a full Kubernetes manifest set (Deployment, Service, ConfigMap, Secret template, StatefulSet for Postgres)

Challenges we ran into

GitHub's MCP server replies using Server-Sent Events framing (event: message\ndata: {...}) instead of a plain JSON body — we had to parse and strip that framing manually before decoding the JSON-RPC response.
MCP tools are method-based rather than one-tool-per-action — pull_request_read alone handles get, get_files, and get_check_runs depending on the method argument passed in, which took real trial and error against the live server to discover.
The GitHub collaborator-permission API doesn't include the repo owner in the collaborator list, so our authorization check needed an explicit owner-check fallback before granting action permissions.
Designing the ambiguity boundary for Layer 7 carefully, so the LLM is a strict enhancement and never a dependency the core pipeline relies on to function.

Accomplishments that we're proud of

A complete, working GitHub-to-Slack-to-GitHub loop: real webhook in, real triage, real Slack interactivity, real GitHub review out — proven live, end to end.
A genuinely production-hardened backend: retries, dead-letter queue, Prometheus metrics, health probes, and both Docker and Kubernetes deployment paths — not just a hackathon prototype.
A local-LLM integration with zero external billing or quota dependency, keeping the whole system self-contained and reliable.
Clean architectural boundaries that made every layer independently testable, one at a time, without breaking what came before it.

What we learned

How to build a real MCP client from scratch, including SSE parsing and method-based tool schema discovery.
How to design a system where AI assistance is additive, never load-bearing — so a model outage degrades gracefully instead of breaking the product.
Real production-reliability patterns: idempotency via unique delivery IDs, bounded retries with backoff, and dead-lettering instead of infinite retry loops.