DecisionLog

An AI Slack agent that auto-detects team decisions in conversation, drafts them as one-click cards, archives them to a database, and answers questions about past decisions via semantic search.

Comment

Inspiration

Every team makes dozens of decisions in Slack — "let's ship Friday instead of Monday," "we're going with Postgres," "dropping dark mode for v1." Then two weeks later someone asks "wait, why did we decide that?" and nobody remembers. The decision is buried in a thread, unsearchable and un-owned. We wanted an agent that quietly watches conversations, recognizes when a real decision happens, and turns that moment into a permanent, searchable record — with zero extra effort from the team.

What it does

DecisionLog is a Slack agent that:

  • Detects decisions automatically. As people chat, it reads messages (with surrounding thread context) and uses an LLM to distinguish a genuine decision from a question, brainstorm, or joke.
  • Drafts a decision card. When it finds one, it posts an interactive Block Kit card summarizing What / Why / Who — resolving relative dates ("Friday" → an absolute date) and pronouns ("I" → the sender's name).
  • Logs with one click. Hit ✅ Confirm & Log and the decision is saved to a Postgres database along with a semantic embedding; ❌ Dismiss throws it away.
  • Answers questions about the past. Mention the bot — "@DecisionLog why did we move the release?" — and it uses semantic (meaning-based) search over every logged decision to answer, even when your wording shares no keywords with the original.
  • Surfaces related discussions across channels via a Slack search integration, so a new decision links back to prior conversations on the same topic.
  • Stays quiet when it should. Questions, hedging ("yeah maybe"), and sarcasm are correctly ignored — no notification spam.

How we built it

  • Slack Bolt for Python (Socket Mode) as the app core — event listeners for messages and mentions, interactive Block Kit cards, and button actions.
  • LLM decision detection with a carefully engineered prompt and a Pydantic-typed structured output schema (is_decision, summary, rationale, decision_maker), so the model returns clean, validated JSON every time.
  • A provider-agnostic LLM abstraction. All app code imports from a single llm module; a one-line LLM_PROVIDER change in .env swaps between Google Gemini and OpenAI with no code edits — prompts and schema are shared so behavior stays identical.
  • Semantic search using embedding vectors + cosine similarity. Each decision is embedded on save; questions are embedded at query time and ranked by meaning, not keywords.
  • PostgreSQL (Neon) via SQLAlchemy for durable storage, with the embedding stored alongside each decision.
  • MCP server integration + Slack's Real-Time Search API to enrich decisions with related cross-channel discussions.
  • FastAPI wrapping the app with a health-check endpoint, async lifespan management, and background task dispatch so Slack's 3-second event window is never missed.
  • Evaluation harnesses for both detection accuracy and semantic-ranking accuracy, so quality is measured, not guessed.

Challenges we ran into

  • Telling decisions apart from noise. The hardest part wasn't detecting decisions — it was not flagging questions, "+1"s without context, and sarcasm. We solved it by feeding recent thread context into the prompt and writing explicit rules, then measuring the result on a labeled eval set.
  • Hidden SDK errors. The Gemini SDK wrapped 429 rate-limit errors inside a tenacity.RetryError, hiding the real cause. We wrote a retry wrapper that unwraps the underlying exception and applies exponential backoff.
  • Slack's 3-second rule. Slow LLM calls risked timing out and triggering duplicate event retries. We dispatch all heavy work to background async tasks and acknowledge Slack instantly.
  • Keeping two LLM providers behaving identically. Sharing one prompt/schema layer across Gemini and OpenAI — and making failures (like a missing embedding) degrade gracefully instead of breaking decision logging — took careful interface design.

Accomplishments that we're proud of

  • 14/14 (100%) accuracy on our decision-detection eval and 4/4 on semantic-search ranking.
  • A genuinely pluggable LLM layer — swap Gemini ↔ OpenAI with one env var.
  • Semantic Q&A that answers "why did we move the release?" correctly even with zero shared keywords with the logged decision.
  • A resilient design where embedding or search failures never break the core log-a-decision flow.

What we learned

  • Structured output beats prompt-and-parse. Typed schemas (Pydantic) made LLM responses reliable enough to build a product on.
  • Context is everything for classification — the same message ("+1") is or isn't a decision depending on what came before it.
  • Evals turn "seems to work" into "works." Writing test cases for detection and ranking caught regressions we'd never have spotted by hand.
  • Design for the provider you don't have yet — abstracting the LLM early made adding a second provider trivial.

What's next for DecisionLog

  • Decision digests — a scheduled summary of the week's decisions posted to a channel.
  • Edit-before-log — let users tweak the summary in a modal before confirming.
  • Richer cross-channel linking and a searchable web dashboard of the decision archive.
  • More providers and local models, plus per-workspace tuning of the detection prompt.

Built With

Share this project:

Updates