Inspiration Every Discord moderator knows the pain: you're managing a gaming server, someone types "you're killing it!" and your keyword filter flags it as a death threat. Meanwhile, actual harassment slips through because bad actors learned to spell "idiot" as "1d10t."
Traditional moderation is broken. Keyword filters are too dumb. Human moderators can't scale. And when a raid hits—dozens of accounts flooding your server with spam—you're already overwhelmed before you can react.
I wanted to build something smarter: an AI moderator that actually understands context, processes messages in real-time, and learns what's normal for your community.
What I Built Discord AI Moderator is a real-time content moderation system powered by Confluent Cloud (Kafka + Flink) and Google Vertex AI (Gemini 2.0 Flash).
The core insight: not every message needs AI analysis. So I built a 4-tier pipeline:
Message → Tier 1 (Banned Words) → Tier 2 (Heuristics) → Tier 3 (AI) → Tier 4 (Periodic Review) FREE, instant FREE, instant Smart Catches patterns Tier 1: Instant banned word/pattern matching (free, <1ms) Tier 2: Heuristic scoring—caps, slurs, threats, mass mentions (free, <5ms) Tier 3: Gemini AI analysis with full context (only ~10% of messages reach here) Tier 4: Periodic conversation review to catch harassment patterns that slip through The result? 80-90% reduction in AI API costs while catching more nuanced violations than keyword filters ever could.
How I Built It Stream Architecture: Every Discord message flows through Confluent Kafka. This decouples ingestion from processing—if the AI is slow, messages queue up instead of dropping. Kafka's partitioning lets me scale horizontally when needed.
Real-Time Aggregation with Flink SQL: Flink SQL handles windowed computations that would be painful to implement manually:
Message rate per user (spam detection) Join rate per minute (raid detection) New account flagging (accounts < 7 days old) Context-Aware AI: Gemini doesn't just see one message—it gets:
Server-specific rules injected into the prompt Server context ("this is a gaming server where trash talk is normal") User reputation history The ability to escalate actions for repeat offenders Periodic Review: Every 3 minutes (or 75 messages), the AI reviews the full conversation buffer. This catches:
Harassment patterns that build over time Dogwhistles and coded language Coordinated attacks from multiple users Escalating toxicity that individual messages miss Challenges The Duplicate Action Bug: During testing, I noticed the bot was timing out the same user 5 times for one offense. The periodic review was re-analyzing messages that had already been processed. Fix: added a reviewed flag to each message so they only get analyzed once.
Kafka Consumer Rebalancing: Running the bot locally while Cloud Run was also running caused constant consumer group rebalancing—both instances fighting over the same partitions. Lesson learned: one consumer group, one deployment.
Prompt Engineering: Getting Gemini to return consistent JSON was tricky. Early versions would sometimes return markdown-wrapped JSON or add commentary. The fix was explicit formatting instructions and regex extraction as a fallback.
Cost Optimization: Analyzing every message with AI would cost a fortune at scale. The tiered system was essential—heuristics filter out 85% of messages before they ever hit the AI.
What I Learned Stream processing changes everything. Kafka + Flink let me build features (raid detection, spam scoring) that would be incredibly complex with traditional request/response architecture.
AI is a scalpel, not a hammer. Using AI for every message is wasteful. The tiered approach—cheap filters first, AI only when needed—is the right pattern for production systems.
Context is king. The same words mean different things in different communities. Injecting server-specific rules and context into the AI prompt made the moderation dramatically more accurate.
Real-time is hard. Debugging streaming systems requires different mental models than REST APIs. Consumer lag, partition rebalancing, and exactly-once semantics all matter.
Built With
- confluent
- discord.js
- flink
- gemini
- google-cloud
- node.js
Log in or sign up for Devpost to join the conversation.