Settled

The decision layer your team and your agents check before they act. Settled tracks what's binding in Slack and answers agents over MCP — so nobody ships on a decision the team already reversed.

Comment

Inspiration

Teams run AI agents in Slack now, and an agent acts on whatever it reads — including a decision the team already reversed. A team picks Postgres, switches to Mongo, then lands on Aurora. A few weeks later the thread is buried, and an agent (or a new teammate) rebuilds on Postgres. Days lost. Decisions don't live in a database; they live in the conversation, and nobody can tell which one still stands. We wanted the thing an agent checks before it acts.

What it does

Settled is a Slack agent that keeps a decision ledger with epistemic status and exposes it to other agents over MCP.

  • Watches channels and detects decisions behind a precision-first confidence gate (uses thread context, catches multiple decisions per message).
  • Asks a human to confirm. A ✅ ratifies, a ❌ dismisses — only a human makes a decision binding, because a wrong "settled" is worse than silence.
  • Tracks status over time: proposed → contested → settled → superseded, each anchored to a verbatim quote + permalink, never a paraphrase.
  • Answer anywhere: /settled <topic>, ask the assistant in natural language, or open the App Home dashboard (with inline Ratify / Details buttons).
  • Runs a decisions:// MCP server, so any external agent — Claude, an IDE bot, your CI — can ask "is this still binding?" before it acts. That's the part a decision log can't do.

How we built it

  • Slack: Bolt for Python, Socket Mode (no public URL). Events API, /settled command, interactive Block Kit (Settle / Dismiss / Details buttons + a decision-detail modal), App Home, and the Assistant surface.
  • Extraction: a cheap noise gate → an LLM classifier (DeepSeek via OpenRouter) returning a confidence score and a verbatim anchor. Precision held by the gate + the human ✅ loop.
  • Ledger: SQLite with an append-only, hash-chained audit log you can verify live (verify_audit_log), toward EU AI Act / DORA traceability.
  • MCP server (decisions://): built on the official MCP SDK; tools is_binding, query_decisions, verify_audit_log + resources, over stdio and streamable-HTTP.
  • 29 tests + CI, deployed always-on on Railway (it self-checks its LLM path on boot).

Challenges we ran into

Precision over recall was the whole game — one wrong "settled" destroys trust, so the confidence gate plus a mandatory human ✅ matter more than catching every decision. Topic-matching was subtle: naive matching wrongly superseded unrelated decisions that happened to share a keyword (e.g. "use Postgres" colliding with "Postgres backups nightly"), so we added guards and tests around it. And keeping the anchor verbatim — never a paraphrase — meant refusing to record a decision rather than fabricating a quote when the model's span didn't match the source.

Accomplishments that we're proud of

  • The guardrail, running for real: python -m demo.agent_coding_guardrail — a coding agent writes a Postgres migration, checks Settled over MCP, sees the team ratified Aurora, and rewrites its own migration before merge, emitting both files.
  • Measured impact: across 8 reversed-decision threads, a status-blind agent acts on a stale decision an expected ~60% of the time; Settled's is_binding, a measured 0% (python -m bench.benchmark).
  • Breadth: a CI gate, an IDE assistant, and a coding agent all consult the same decisions:// ledger (python -m demo.breadth).
  • A tamper-evident hash-chained audit log with a live verifier, 29 passing tests, CI, and an always-on deploy.

What we learned

The ceiling for organizational AI is epistemic, not retrieval, fidelity — knowing which decision still binds matters more than finding the text (echoing Retrieval Is Not Enough, Bottino et al., 2026). And keeping the verbatim source beats storing summaries for preserving a decision's exact constraints (Verbatim Chunks Beat Extracted Artifacts, Tao An, arXiv:2601.00821) — which is why every decision keeps its quote and permalink. Precision-first design, with a human in the loop, is what makes an agent-facing source of truth trustworthy.

What's next for Settled

Org-chart-aware ownership, embeddings for topic linking, connectors (wiki / Jira / HR) feeding context, and a public MCP endpoint so any team's agents can consult their own decision layer.

Built With

  • block-kit
  • deepseek
  • mcp
  • model-context-protocol
  • openrouter
  • python
  • railway
  • remotion
  • slack
  • slack-bolt
  • socket-mode
  • sqlite
Share this project:

Updates