Triage

• 

• 

• 

• 

  Agent Output

• 

Inspiration

During disaster response, the bottleneck isn't willingness — it's routing. When a hurricane or earthquake hits, volunteer Slack workspaces get flooded with thousands of unstructured messages, and existing automation tools like Zapier or Slack Workflow Builder rely on rigid keyword matching. A message like "Road on Route 4 is washed out, need a way to get insulin to the shelter" has no obvious trigger word, but it's a critical, time-sensitive, multi-part request. Under keyword routing, it just sits in #general until a human happens to scroll past it. We wanted to build something that reads intent the way an experienced dispatcher would — by asking "has something like this happened before, and what worked?" instead of "does this message contain the word 'medical'?"

What it does

Triage is a Slack agent that reads incoming messages for intent and severity, not keywords. It searches historical resolution threads using Slack's Real-Time Search (RTS) API to find precedent for how a similar request was handled before and by whom, then cross-references a volunteer availability database via a custom MCP server to confirm the right responder is actually active right now. It surfaces all of this as a single, actionable Block Kit card to a dispatcher — a severity tag, the matched precedent, and a one-click route. If the historical match is weak, Triage doesn't guess: it explicitly flags "Human Triage Required" instead of auto-routing. There's also no plain "Dismiss" button — every path off a message requires either a routing target or a documented reason, so nothing critical quietly disappears.

How we built it

• Slack Bolt (Python/JS) for event handling, interactivity, Block Kit rendering, and modals
• Slack RTS API, scoped strictly to channels the app is invited to, for historical precedent search
• A custom MCP server for volunteer availability lookup (active/resting status)
• A Vercel serverless function calling the Anthropic API for intent, severity, and confidence extraction
• Postgres/Airtable as the volunteer availability store

Confidence scoring is derived from semantic similarity between the incoming message and matched historical threads, deliberately not raw LLM self-reported confidence, since that's known to be poorly calibrated. The 80% routing threshold is exposed as a config value (CONFIDENCE_THRESHOLD) rather than hardcoded, so it can be tuned per deployment.

Challenges we ran into

Slack's native Agent Builder synthesis step looked like the obvious place to inject our RTS and MCP context — but on our Day 1 technical spike, we couldn't find a documented API for feeding external context into that native step. Rather than force a workaround into an undocumented pathway, we built our own synthesis layer (Vercel + Anthropic API) and used standard Bolt methods to render the resulting UI. It's a contained decision — if Slack documents that capability later, migrating the synthesis step doesn't touch our RTS, MCP, or Block Kit layers. We also had to be disciplined about not overselling what was live vs. mocked: the MCP server itself is a real, functioning integration, but the volunteer records behind it are seeded synthetic data for this prototype, not a live registry — and we decided early to disclose that plainly rather than blur the line in the demo.

Accomplishments that we're proud of

We're proud that Triage refuses to fake confidence. A dispatcher-facing tool that guesses wrong in a crisis is worse than one that says "I'm not sure" — so low-confidence matches never auto-route, and the one truly destructive action (Resolve/Mark Duplicate) requires a mandatory modal with a reason, so it can't be triggered by an accidental click. We also built a benchmark methodology with a scoring rubric decided before labeling, to avoid post-hoc rule-bending — and we're committed to reporting whatever run_eval.py actually outputs, even if it's a modest improvement over keyword baseline rather than a dramatic one.

What we learned

Verifying integrations against real documentation before committing to an architecture saved us from building on assumptions we couldn't defend in a demo — the Slack native synthesis decision is the clearest example. We also learned that for a tool built for crisis response, the most trustworthy thing we could do wasn't polish every rough edge — it was label them clearly: what's live, what's mocked, and what the confidence threshold actually means. A prototype that's honest about its edges is more useful to a real dispatcher than one that hides them behind a smooth demo.

What's next for Triage

• Replace seeded volunteer data with a live registry integration
• Migrate the synthesis step to Slack's native AI pipeline if/when it supports external context injection
• Add multi-language support for international disaster response
• Make severity taxonomy per-workspace tunable
• Re-label the benchmark set with multiple independent annotators to validate the routing accuracy numbers

Built With

• claude
• cuda
• mcp
• openrouter
• python
• real-time
• slack