Tea

Tea scans your CRM deals, flags what's at risk, and cites the exact Slack message for every claim. Can't back it up? It stays silent. We spill the tea, with receipts.

Comment

Inspiration

Every "AI deal-risk" tool tells you a deal is in trouble. None of them can prove it. You get a confident paragraph of LLM rationale and no way to check whether it's real or hallucinated. In a category that sells the word "evidence-backed," the evidence is usually a black box.

Before writing a line of code, I went digging, and two things stood out. First, the commercial category (Gong, Clari, People.ai, Agentforce in Slack) all claim "evidence-backed," but when you look closely they ship citation existence, not citation support: a link, with no guarantee the linked message actually backs the claim. Second, the research community already knows how to measure that difference. Work like ALCE (2305.14627), FActScore (2305.14251), and Attributable-to-Identified-Sources (AIS, 2112.12870) frames faithfulness as "does the cited source actually entail the claim," while self-consistency (2203.11171) and multi-agent debate (2305.14325) show how to make a model check its own work. The gap was sitting in plain sight: the rigor exists in the literature, but nobody had put it behind a working Slack agent.

So I built the opposite of a black box: an agent that would rather stay silent than guess. If it can't point at the exact Slack message behind a claim, it doesn't make the claim. That discipline turns out to be the hard part nobody ships.

What it does

Tea is a Slack agent that scans your open CRM deals, searches your team's Slack history for what's actually happening on each one, and flags what's at risk. The rule it never breaks: no evidence, no flag.

Every risk it raises cites a real, clickable Slack permalink. Before anything reaches you, each proposed flag runs through a faithfulness gate:

  1. Provenance. The citation has to be a real message (or a real CRM field value), verified by exact set-membership, not vibes.
  2. Support. The cited message actually has to entail the claim. A flag whose "evidence" is real but off-topic gets dropped.
  3. Cross-examination. Tea searches the deal's own chatter for messages that refute its own flag, and withdraws the flag in public if it finds one. It argues against itself before it speaks.

You watch this happen live. The scan streams each stage, so you see flags get dropped and withdrawn in real time, then it collapses into a clean verdict card with a full evidence dossier behind it.

It's more than deal risk. @Tea ask <anything> runs the same cite-or-refuse gate over your whole channel corpus, and a weekly "Tea Time" briefing surfaces what moved. Deal risk is just the first vertical. Underneath it is a faithfulness layer for answering any question over private conversations. Tea also exposes itself over MCP, so the same gate is portable to Claude Desktop, Agentforce, or any agent that wants a trust primitive.

How I built it

  • Slack Bolt on Socket Mode for the agent surface, with @slack/web-api streaming (chatStream) for the live gate.
  • Slack's Real-Time Search API (assistant.search.context) as the only source of Slack evidence. Every citation traces back to a permalink it returned.
  • A pluggable CrmAdapter interface that fuses CRM state with Slack evidence before any flag is scored. A Salesforce-shaped mock adapter drives the demo, and a real Salesforce adapter drops in behind credentials.
  • Claude Sonnet for scoring, Claude Haiku as the fast judge for the entailment, sub-claim decomposition, and cross-examination passes.
  • A post-LLM citation validator in plain deterministic TypeScript. This is the moat. The model proposes; the validator disposes.
  • libSQL/SQLite (WAL) for the deal, score, and audit store, and MCP to expose the gate as portable tools.

Challenges I ran into

  • Correct is not faithful. Early on the validator only checked that a citation existed. The real frontier is whether the cited message supports the claim. That became the entailment layer and reframed the whole product.
  • Tea citing Tea. Tea posts its verdicts into the channel, and the search API retrieved those posts as "evidence" on the next scan. The fix matches Tea's own surface text at ingest.
  • Making abstention demoable. The one thing the category structurally can't show is an agent refusing to answer when the evidence isn't there. Making that refusal a first-class on-screen moment, not an error, was the design challenge.
  • Streaming to justify the wait, not hide it. Everyone streams to feel fast. Tea streams to make the diligence visible: the deliberate pace is the pitch.

Accomplishments that I'm proud of

I built a reproducible, deterministic eval that proves the gate works, committed to the repo as EVAL.md. Across 17 labeled deals the scorer proposes 32 flags, and only 15 of them survive the full faithfulness gate, a 47% survival rate (reported as ALCE citation precision). The other 17 are removed, and the breakdown is the whole story:

  • 10 fail provenance (5 with an unresolvable permalink, 5 with a CRM value that does not match),
  • 3 are cited but not entailed by the message,
  • 1 more when sub-claim decomposition finds an atomic claim the citation does not support,
  • 2 are withdrawn after cross-examination surfaces counter-evidence in the deal's own chatter,
  • 1 is dropped on self-consistency when the scorer does not reproduce it across samples.

At the deal level, the survivors change the risk band on 12 of the 17 deals, and in 5 of those the verdict collapses from a real risk to none. Every flag that survives is cited and supported: citation recall and supported-claim ratio both 100% (ALCE, FActScore).

And because 100% over a finite sample is an estimate rather than a guarantee, live runs report each rate with a 95% Wilson lower bound instead of pretending certainty. The eval aborts if any keep, drop, or withdraw decision ever disagrees with its ground-truth label, so the published numbers cannot quietly drift.

What I learned

The hard, valuable part of a trustworthy AI product is everything the model doesn't say. Constraining an LLM, forcing it to prove, refute itself, and abstain, is more engineering than prompting. And "we don't guess" only earns trust if you also refuse to overstate your own accuracy, which is why Tea reports what it can prove (deterministic provenance) differently from what it can measure (statistical precision with a confidence bound).

Why now

The timing isn't an accident. A "verifiable AI agent" category is forming in 2026, and it's Salesforce's own thesis: Slack as the front door to the agentic enterprise, with Slackbot orchestrating a fleet of agents over conversational context. Salesforce's own engineering writes about verifiable citations, but the shipped surface stops at a clickable link. At the same time the pressure to prove an AI's reasoning is turning into regulation and cost: FINRA is pushing audit trails for AI decisions, the FDA treats hallucination as a device risk, and analysts have started naming the "hallucination tax" and a coming "veracity economy." When every agent can generate an answer, the scarce thing is a receipt. Tea is built for exactly that: an agent whose output you can check, in the one place work already happens.

What's next

From a deal tool to a trust layer. The same faithfulness gate that scores deals can answer any question over any private corpus, and it belongs to more than one agent: exposed over MCP, it becomes a portable "verify before you speak" primitive that any Slack or Agentforce agent can inherit, returning a citation-precision score with every answer. Deal risk is vertical one. The receipts layer underneath it is the product.

Built With

Share this project:

Updates