-
/triage in #all-rag: the agent returns priority, severity, components — and explains why it chose P2, grounded in real issue history.
Inspiration
Every engineering team on Slack pays the same invisible tax: manual bug triage.
Reports land in a channel as noise — "checkout is broken??", a screenshot, half a stack trace. Then a human has to rewrite it into something a developer can act on, judge how urgent it is, dig through the issue tracker to see if someone already reported it, and file it. It takes minutes per report, it happens dozens of times a week, and it still goes wrong: duplicates get fixed twice, and real P0s get buried under noise.
We didn't want to build another dashboard that people would have to remember to open. We wanted to delete the triage step entirely — at the source, in the tool the team already lives in. So we built an agent that lives inside Slack.
What it does
Paste any messy bug report into Slack. The agent instantly returns an interactive triage card:
A clean title, summary, and reproduction steps Priority (P0–P3) and severity — each with a stated reason, not a guess Expected vs. actual behavior Duplicate candidates, matched against your team's actual past issues A File to GitHub / Plane button that creates the real issue in one click — the card then updates to It also drafts release notes from resolved issues with /release-notes.
There are four ways to reach it, and all of them stay in Slack:
You do this It does this /triage Posts an interactive triage card to the channel Hover a message → Triage this Triages that exact message in a thread @BugTriage Triages in a channel thread Open the AI Assistant panel and paste Chat-style triage, replies in-thread How we built it The whole thing is a Python service that speaks Slack on one end and a real retrieval stack on the other.
The surface — Slack Bolt over Socket Mode. No public URL, no ngrok, no inbound webhooks. The app opens a WebSocket to Slack and handles the slash command, the @mention, the message shortcut, the native AI Assistant panel, and the Block Kit button clicks. The triage card is Block Kit, and the File button mutates the card in place so the "filed" state lives right where the conversation is.
The brain — an agentic RAG loop on Claude Opus 4.8. This is not a single prompt. The model runs a genuine tool loop: it decides for itself what to search for, issues its own queries against the issue history (expanding and re-querying when the first search is thin), and only then calls a strict submit_triage structured-output tool that ends the loop. That means the duplicate detection is the model's own investigation, not a cosine-similarity threshold bolted on afterward — and the card shows the search trace it took to get there.
The retrieval — real hybrid search. Dense vector ANN over Qdrant (HNSW) plus BM25 keyword scoring, fused with Reciprocal Rank Fusion. Semantics catch "cart fails at payment" ≈ "checkout 500"; BM25 catches the error codes and identifiers that embeddings smear away. RRF gets us both without tuning a weight.
The knowledge base is real. You ingest an actual repo — its GitHub issues and its docs — through a heading-aware overlapping chunker into Qdrant. Our demo KB is 475 chunks from a live private repo. There is no seed data and no mock: every priority call and every duplicate the agent flags is grounded in your team's own history.
The engine is also an MCP server. python -m app.mcp_server publishes the same engine over the Model Context Protocol with four tools — triage_report, search_issues, file_issue, release_notes — so Slack's agent runtime or any MCP client can call it directly.
Everything is pluggable. LLM providers fall back down a chain (Anthropic → any OpenAI-compatible endpoint → local Ollama, free and keyless), and embeddings do the same (Voyage → Ollama → an offline lexical tier). The app boots and works even with zero paid keys.
Required technologies — we use two of the three: Slack's AI capabilities (the native AI Assistant panel: suggested prompts, live "triaging…" status, threaded replies) and MCP server integration.
Challenges we ran into Making dedup trustworthy. Our first version embedded the report, took the top-k neighbors, and called anything above a threshold a duplicate. It was confidently wrong constantly — near-miss issues look very similar in vector space. Two things fixed it: hybrid retrieval (BM25 pulled in the exact error strings dense search was blurring), and handing the search to the model as a tool so it could re-query, compare, and justify the match instead of trusting one similarity score.
Slack's interaction budget. Slack wants an acknowledgement within 3 seconds; a real agentic loop takes longer than that. We had to split every entry point into an immediate ack ("⏳ Triaging…") and a deferred update that swaps in the finished card — and then make the File button idempotent, because a button in a channel will get clicked twice.
Getting structured output that never breaks the card. A malformed field means a broken Block Kit render in front of the whole channel. Forcing the model to terminate through a strict Pydantic-validated submit_triage tool — rather than parsing prose — made the output contract non-negotiable.
One process, one lock. Embedded Qdrant locks its store to a single process, which is a quiet trap when you're running the web console and the Slack agent side by side during a demo.
Accomplishments that we're proud of The agent shows its work — you can see the searches it ran to reach its conclusion. One click, real write. The File button creates a genuine GitHub issue (private repos included) or a Plane work item — not a simulated one. It runs fully free if you want it to — local Ollama and a local vector DB, no API keys required. Four Slack surfaces, one engine — and that same engine ships as an MCP server. What we learned Grounding beats model size. The single biggest jump in output quality came from feeding the agent real issue history, not from a better prompt. Let the model search. Retrieval as a tool the model drives beat retrieval as a fixed pre-processing step, by a lot. Hybrid isn't optional. Pure vector search silently fails on identifiers, error codes, and version numbers — exactly the tokens bug reports are made of. The last mile is the product. Being in Slack, with a button that files the issue, is what turns a good triage model into something a team would actually keep using. What's next for AI Bug Triage Proactive triage — the agent watches a #bugs channel and triages every new report unprompted. Weekly release-note digests posted straight to a channel. Learning from corrections — when a human overrides a priority, feed that back into the KB. Slack Marketplace distribution.
Log in or sign up for Devpost to join the conversation.