COMPILER — Decision Review Agent for Slack

Compiles which validaets against decision

Inspiration

Teams review every code change before it ships — but the decisions that cause those changes ("launch in Europe in Q3", "migrate the database in August") get made in a Slack thread with zero checks. The context that would catch a problem already exists in Linear, GitHub, Notion, and old Slack threads; nobody has time to cross-reference it. COMPILER treats a decision the way a compiler treats code: parse it, type-check it against reality, and report what's broken before it executes.

What it does

Run /compile <decision> (or @mention COMPILER in a thread). The agent:

Parses the plain-English decision into a typed structure — actors, objective, timeline, dependencies, assumptions, resources, expected outcomes.
Runs 5 checkers in parallel against connected systems:
- Dependency Checker — engineering capacity and tracked tickets (Linear), infrastructure/CI signals (GitHub), compliance docs like GDPR readiness (Notion).
- Type Checker — contradictions against real milestones and prior decisions (e.g. "ship in August" vs "engineering freeze until September"), with a hallucination guard that drops any conflict whose cited commitment doesn't match a retrieved record.
- Scope-Leak Detector — implicit downstream work nobody planned, cross-checked against Notion/Linear so only genuinely untracked work is reported.
- Risk Analyzer — technical/legal/timeline/infra risks.
- Related-Context Checker — Slack's Real-Time Search surfaces prior threads about the same topic as linked evidence.
Compiles a report with a hard trust rule, enforced in one place no checker can bypass: only findings confirmed by a real record become blockers or gaps. Everything the model merely inferred goes to a separate "NEEDS HUMAN REVIEW" section. COMPILER never approves or blocks — it surfaces evidence for humans.

Reports are private-first: only the person who ran /compile sees it, with a "Share to thread" button — nobody's decision is publicly graded without consent. Action buttons on each report: Create Linear tickets for confirmed gaps (never for inferred ones), Mark reviewed (persisted, message updated in place), and Re-compile.

How we built it

Slack agent — Bolt (TypeScript/Node), Block Kit reports, slash command + mentions + optional passive decision detection (off by default, ephemeral nudge only).
MCP connector layer — the single boundary between checkers and the outside world: Linear (GraphQL), GitHub (REST), Notion (Search API), Slack RTS (search.messages). With no keys it serves seeded mock data so the whole pipeline demos with zero credentials; with keys it goes live — checker code never changes.
Model-agnostic LLM layer — one LLMProvider interface with four adapters (Claude, OpenAI, Gemini, local Ollama); switching models is an env var, not code.
Multi-tenant & security — Slack OAuth v2 installs per workspace; /compile connect links each provider per channel. GitHub uses a GitHub App with on-demand installation tokens (no long-lived credential stored); Linear/Notion tokens are AES-256-GCM encrypted at rest; connect links carry HMAC-signed state. All lookups are scoped to the tenant and channel, so a decision compiled in one channel never leaks into another's report.
Decision History — every compile is stored (Postgres, or in-memory for single-workspace) and fed back into the Type Checker, so COMPILER gets smarter with every decision it sees.

Challenges we ran into

The hardest problem wasn't technical — it was trust. An LLM will confidently invent a "blocker," and one false ERROR in front of a VP kills the product. Our fix is structural, not prompt-based: every finding carries a confidence field (confirmed = grounded in a retrieved record, inferred = model judgment), and the report compiler routes anything inferred or unevidenced to NEEDS HUMAN REVIEW — centrally, so no checker can bypass it. Risk findings are capped at WARNING. Even an empty report says "absence of findings is not approval."

Second: consent and isolation. Auto-reviewing people's messages publicly is creepy, so reports are private-first, passive detection is opt-in and author-only, and RTS results from private channels/DMs are dropped so the installing user's private matches are never quoted to others.

What we learned

Honesty about confidence beats confident-sounding output. Separating "we found a record" from "the model thinks" is the whole product.
Graceful degradation matters: checkers run under Promise.allSettled, so one failing checker never sinks the report.
Channel-scoped credentials are the right unit for org tools — a connection made in #product-launch shouldn't power compiles in #random.

What's next

pgvector embeddings for similarity-based type-checking against prior decisions, connector response caching, more connectors (Jira, Confluence, Drive) feeding the same evidence substrate.