Inspiration
I've watched too many teams get burned by messages that just sound right. Someone drops a confident "our refund window is 90 days" into a thread, a few people plan around it, and nobody notices the real policy says 30. Or a decision gets made on a Tuesday afternoon while the two people it actually affects are asleep in another timezone. Slack moves fast, and that speed hides how little of it anyone is really checking.
Every Slack agent I looked at was built to do more work, faster. Summarize this, draft that. But my problem was never a shortage of output. It was trust. I wanted something that reads along quietly and helps me figure out what actually deserves it. So, I built the opposite of a task bot.
What it does
Arbiter is one bot with six kinds of judgment, and you trigger each with a normal message:
- Check a claim, and three different models argue it out, grounded in the web and in your own workspace history.
- Paste a long, polished, empty update, and it quietly tells you (only you) how little is actually in it.
- Make a decision in a thread, and it pulls in the people who aren't there but should have a say, quoting what they really said.
- Ask what a teammate thinks while they're away, and it answers only from their own messages, with links, and flat out refuses if they never touched the subject.
- Name a few people, and it has them talk a question through in their own voices and lands on where they'd end up.
- Come back from a day off, and it tells you what you missed.
The thread running through all of it: it never makes anyone up, and it can always show its work.
How I built it
It runs on Python and Slack Bolt over a websocket, with a small state machine deciding which kind of judgment a message needs before any expensive model runs. Most messages get handled for free by cheap heuristics and never hit a model at all. The ones that do go to one pipeline, never five at once, so it doesn't dogpile a single message.
It leans on all three of the challenge technologies because it genuinely needs them: real-time search to pull the actual messages behind a fact-check or a person's opinion, MCP to reach back into Slack and flag things or export an audit trail, and the assistant pane for quick prompts. A graph database remembers claims, who said what, and which alerts I've told it to be more careful about, so a thumbs down actually changes its behavior next time.
Challenges I ran into
The delegate was the hardest part, and not for technical reasons. Making an AI that talks like your coworker is easy. Making one that refuses to is not. The whole thing falls apart the second it guesses someone's opinion, so I put a second model in front of every answer whose only job is to check that every sentence traces back to a real quote. If it can't, the delegate says nothing and pings the actual person instead.
The other honest one was routing. Getting a message to the right kind of judgment sounds trivial until you actually measure it. I wrote a benchmark of 150 labeled cases, hit 98 percent, and then got suspicious, because I had tuned it on those same cases. So I wrote a second set of messy, real-sounding messages I had never looked at, and the score dropped hard. That gap was the most useful thing I found all week. It showed me my decision detection was leaning on a keyword list instead of actually understanding the message, and I rebuilt it to trust the model instead.
Accomplishments that I'm proud of
Honestly, running the whole thing on nothing. I'm one person with no budget, so it's held together with free model tiers across seven providers (with fallbacks for when they rate limit me), a free graph database, and a free host I keep awake with a pinger. It still feels live.
And the numbers hold up where it counts: fact-checking beats a single strong model by ten points, the workslop detector catches 20 out of 20, and there are 66 tests so I don't quietly break it.
What I learned
The hard part of an agent isn't making it capable. It's making it hold back. The features I'm proudest of are the ones where Arbiter says no. It won't guess a person's position, it won't call a vague message substantial, it won't flag something it can't back up. Restraint turned out to be harder to build than capability, and it's the entire reason you'd trust the thing.
Log in or sign up for Devpost to join the conversation.