Slack_MiniEval

MiniEval doesn't add another AI to your Slack - it audits the AI already there. Right-click any AI summary and it scores how faithfully it matches the original conversation, flagging hallucinations.

Comment

Inspiration

Slack AI can summarize any thread or channel with a single click. It's fast, it's confident, and it's everywhere - thread recaps, channel digests, meeting notes. But there's a problem nobody talks about: nobody checks if those summaries are actually true.

You ask an AI for a quick summary. It sounds authoritative. You forward it to your boss. Later, you find out half of it was wrong - a draft policy read as "approved," a contested deadline reported as "confirmed," a $50K number that somehow became $100K.

There are plenty of hallucination-detection tools for developers - Guardrails, Arthur, LangChain evaluators. But they all live in notebooks and CLIs. None of them are in Slack, where the summaries people actually rely on are being read. We wanted to close that gap: bring AI quality control to the people who suffer from AI mistakes, not just the engineers who build with AI.

What it does

MiniEval doesn't add another AI to your Slack - it audits the AI that's already there.

Right-click any message containing an AI summary (or run '/minieval-check'), and MiniEval:

  1. Fetches the original thread or channel content the summary claims to represent
  2. Runs a Natural Language Inference (NLI) model to score how 'faithful' the summary is to that source
  3. Posts the verdict right in the thread - a ⚠️ "warning card" if the summary is likely hallucinated, or a ✅ "verified card" if it's faithful - with the faithfulness score, model, and latency

Every check feeds a live "Slack Canvas dashboard" showing a workspace trust score and per-channel hallucination rates, and '/minieval-stats' gives an on-demand channel report. The entire evaluation engine is also exposed as a "spec-compliant MCP server", so the same faithfulness checking is callable from any MCP client - not just Slack.

The engine runs locally. No message content ever leaves the workspace.

How we built it

  • 'Slack Bolt (Python), Socket Mode' for the agent - message shortcuts, slash commands, and modals as on-demand entry points

  • "NLI faithfulness scoring using' cross-encoder/nli-deberta-v3-small', treating the source as premise and the summary as hypothesis; entailment probability becomes the faithfulness score

  • "Block Kit" for the warning/verified cards, with score bars and metadata

  • "Slack Canvas API" for the live team-facing trust dashboard

  • "A standalone MCP server" (stdio transport) exposing 'evaluate_summary', 'get_channel_stats', and 'get_recent_evaluations'

  • "SQLite" for evaluation logging and stats

    Challenges we ran into

This project was a series of hard-won debugging lessons:

-"The engine that scored everything 0%." The NLI model returns generic LABEL_0/1/2 outputs instead of named labels. Our first mapping silently defaulted entailment to zero, so 'every' summary came back as a hallucination. It took building an offline batch evaluator to prove the engine worked in isolation - and then realizing the bug was in how the live path read the model's output.

  • "Honest numbers." We caught ourselves almost shipping a dashboard populated by a script that fabricated a hallucination rate with random numbers. We tore it out and rebuilt everything to display only real, reproducible data - even when that meant the dashboard proudly showed "4 evaluations" instead of an impressive-looking fake 500.

  • "Getting the details right" - permission scopes for Canvas, keeping stdout clean for the MCP protocol, resolving a private-canvas quirk, and the very human final-day scramble of sandbox member limits and email typos.

Accomplishments that we're proud of

  • A hallucination detector that a "non-technical user can run with a right-click" - no code, no notebooks

  • Both verdicts working correctly: it catches real hallucinations and verifies genuinely faithful summaries, so it's a trust tool, not a smoke alarm that only ever cries wolf

  • A "real, independently-callable MCP server" - not internal plumbing, but a spec-compliant server any MCP client can connect to

  • A commitment to "honest evidence over impressive-looking fakes", all the way down to reproducible evaluation scripts anyone can re-run

  • Sub-200ms local inference on CPU - no GPU, no external API

What we learned

  • "Faithfulness is about the pair, not the pieces." The same summary can be 99% faithful or 0% faithful depending entirely on what you compare it against. Getting the 'source' right matters as much as the model.
  • "Debug at the source of truth." We wasted time staring at Slack's output when the answer was always in the terminal logs. Once we read what the model actually received and returned, every bug became obvious.
  • "The unglamorous last mile is most of the work." The engine was "done" long before the project was - scopes, permissions, clean data, and paperwork took real effort.

What's next for Slack_MiniEval

  • "Auto-detection" of AI summaries where Slack's API allows it, for zero-click checking
  • "Configurable thresholds and domain tuning" so teams can calibrate strictness per channel
  • "Broader benchmarking" against summarization-specific datasets (XSum, FRANK) for published accuracy numbers
  • "Beyond Slack" - leaning into the MCP server so the same faithfulness layer works across any AI tool a team uses
  • Richer Canvas analytics: trends over time, most-corrected channels, and exportable trust reports

Built With

Share this project:

Updates