TriageOps

Moderation that understands intent, not just keywords, " Automate the obvious, queue the ambiguous"

Comment

Inspiration

Keyword filters are a thing of the past. In the age of AI-generated spam, you can’t filter human intent with a regex script. Bad actors are changing their vocabulary easily to circumvent static filters, masking toxic behaviour and hidden payload behind clean words.

TriageOps was built to address this nuance gap. Instead of searching for trigger words, I use LLMs to understand the contextual meaning of each post. It catches what regular bots miss, intelligently bucketizing content into actionable buckets. TriageOps combines this AI comprehension with a human-in-the-loop dashboard to automate the obvious, queue the ambiguous, and return control to the moderators.

What it does

TriageOps is an AI-powered Reddit moderator copilot. It lives on your subreddit and acts like a hyper-aware assistant that catches things that normal bots miss, while keeping real humans in the driver’s seat.

Here’s exactly how it protects communities:

  • Context-Aware Filtering : TriageOps doesn’t simply scan for the word “crypto” and ban everyone. It understands the conversation. It can instantly distinguish between a user sharing a genuine finance article and a spammer looking to shill a pump-and-dump coin. It will even pick up on toxic, passive aggressive rants that don’t have a single curse word.
  • The Speed (Event-Driven Micro-batching): Most bots have a slow 5 min timer to look for rule breaking posts. By the time the bot wakes up the community already sees the spam. TriageOps uses an event-driven "bucket" system. As soon as some posts come in, the bucket tips over and processes them all at once as a single batch. Save huge amounts on API costs and get lightning fast real-time moderation.
  • The Human-in-the-Loop Queue (The Safety Net): AI isn't perfect, and it shouldn't pretend to be. If a post is pure internet slang (like "skibidi rizz gyatt"), an inside joke, or heavily relies on an image, TriageOps doesn't guess. It safely parks these ambiguous posts in a clean "Human Review Queue," ensuring real moderators always have the final say.
  • Live Cluster Dashboard (The Vision): Moderators get a beautiful, real-time visual dashboard built natively into Reddit. You can watch topological bubbles grow as the AI categorizes incoming posts (e.g., "Spam," "Malware," "Toxic"). It gives moderation teams an instant, visual heartbeat monitor of their community's health and current attack vectors.
  • Enterprise Auto-Failover (The Reliability): Built-in circuit breakers mean your subreddit's protection never goes offline. If your primary Google Gemini API key hits a rate limit or a daily quota during a massive traffic spike, the Vault automatically hot-swaps to your backup keys so the shield never drops.

How I built it

I engineered TriageOps as a fully serverless, event-driven application operating natively within Reddit’s Devvit ecosystem. I wrote the backend in TypeScript, utilized Devvit's built-in Redis for state management, and powered the semantic reasoning engine with Google's Gemini 3.1 Flash Lite API.

  • Event-Driven Micro-Batching (The Backend): To solve the problem of API rate limits, I built a custom pipeline. When a post hits the Devvit webhook, it drops into a Redis memory queue. The system actively monitors its own capacity—the exact millisecond the queue hits the user-defined batch limit, it acts like a tipping bucket and automatically triggers the LLM, guaranteeing real-time moderation while minimizing API calls and also offers setting for real time clustering for individual posts.
  • Bespoke Custom Web Views (The Frontend): Standard UI blocks weren't powerful enough for the analytics I wanted to display, so I leveraged Devvit's Custom Web Views. I built a completely dependency-free, raw HTML/CSS/JS frontend (splash.ts). By bypassing bloated frameworks like React or Vue.
  • Real-Time SVG Topology Math: To visualize the AI clustering, I didn't rely on heavy external charting libraries. I wrote custom 2D grid math that dynamically calculates X/Y coordinates and radii based on live Redis data, injecting raw SVG elements directly into the DOM. This creates a beautiful, floating topological map of community health that scales perfectly to any screen size.
  • The Secure Configuration Vault: The dashboard features a real-time settings vault that communicates directly with Redis. Moderators can dynamically update custom subreddit rules, adjust confidence thresholds, define JSON escalation matrices, and safely inject (and mask) backup API keys without ever needing to touch the backend code or redeploy the app.
  • Human-in-the-Loop & RLHF Infrastructure: I built a dedicated "Review Queue" tab where ambiguous posts are parked. Crucially, I implemented a submitFeedback mechanism (the 👍 Correct / 👎 Incorrect buttons). This logs moderator decisions back into a specialized Redis queue, laying the groundwork for Reinforcement Learning from Human Feedback (RLHF) to fine-tune the LLM prompts in future updates.
  • Frictionless Developer Tools: Because I built this for a high-stakes environment, I engineered custom safety features into the UI itself—including a mandatory 3-second cryptographic-style "unlock delay" on the database wipe button to prevent accidental data loss during high-pressure moderation. ## Challenges I ran into

Devvit Context Injection and Context Routing Split

Here’s what I ran into: Devvit’s handling of dependencies totally depends on how you trigger the runtime. With HTTP endpoints and the usual triggers, you just pull in globals—import { redis, reddit } from "@devvit/web/server"—and everything works. But when you run scheduled background jobs, Devvit flips the script and hands you an event-driven context object, so you have to dig your services out from there instead.

What did this break? Well, anytime I tried to share logic—like processBatchQueue()—between webhooks and background jobs, things blew up at runtime because of undefined properties. So I had to build an explicit dependency picker at the top of our queue processor: const db = jobContext ? jobContext.redis : redis; (and so on for every service). Channeling all database reads and Reddit moderation calls through this layer stopped the crashes. Sure, it made the type system angry and debugging locally became a mess, but at least it worked.

Manual JSON Cleanup with Raw HTTP Fetch

Devvit lives in this strict serverless box, so when I tried to bring in the full @google/generative-ai client SDK, things got heavy—dependencies ballooned and compilation failed. I ditched the SDK and started making bare-bones fetch calls ourselves to the Gemini REST API.

But this created a new mess. Without the SDK, I didn’t have runtime type checking or response schema validation anymore. Gemini’s API liked to wrap its response arrays in markdown code blocks (like json ...), which made JSON.parse() blow up with syntax errors. So, before parsing, I had to hard-strip those wrappers ourselves—just a little responseText.replace(/json/g, "").replace(//g, "").trim()—right in the execution loop. Not pretty, but it kept the pipeline running.

State and Race Conditions with Redis Queues

Our frontend needed a real-time dashboard, but traditional databases were off the table. Instead, I wired up Redis in two parts: a Sorted Set for queuing post IDs in order (flagged_posts_queue), and a Hash Set for each post’s metadata (flagged_posts_data).

But when a moderator approved or removed a post, it kicked off several async cleanup steps in the backend. I kept hitting race conditions—sometimes, the dashboard’s next refresh (via /api/getTriageQueue) would grab stale Redis data before the deletes finished, so ghost items popped back into the UI. The fix? I enforced a tight, atomic order for every deletion step, making sure nothing got left behind while waiting for Reddit’s external API to catch up. It’s fragile, but it kept the ghosts out of sight.

Accomplishments that we're proud of

  • The Micro-Batching Engine: I broke the standard “one API call per post” bottleneck. By batching anywhere from 10 to 50 posts into a single prompt using Redis sorted sets, I chopped down outbound Gemini API requests by at least 90%. That’s what makes large-scale AI moderation actually affordable.
  • Enterprise Auto-Failover (The Vault): I built a sturdy circuit breaker into our main processing loop. If our main Gemini API key hits a 429 RESOURCE_EXHAUSTED error during a traffic spike, the system waits, retries, and if it keeps happening, hot-swaps to our backup keys or even falls back to lighter models like gemini-3.1-flash-lite. I don’t lose a single webhook.
  • The Smart Escalation Matrix: Forget simple “ban or ignore” bots. TriageOps routes infractions based on how severe they are—minor stuff gets filtered away quietly, obvious spam gets nuked automatically, and stuff in the gray area pops up in a live dashboard for a human touch.
  • Zero-Latency Dashboard Integration: I brought all of this into Reddit’s own UI. Real-time human review and analytics pop up natively in the dashboard. No client-side lag, either—all the number crunching stays on the backend.

What I learned

  • LLM Non-Determinism requires isolation layers: You can’t count on an LLM to spit out perfect JSON. Sometimes the input is just plain weird—thanks, Reddit—and backend crashes aren’t an option. So, I built solid sanitization and schema validation to keep things stable.
  • Event-Driven Architecture is Mandatory: Subreddit traffic doesn’t flow smoothly; it spikes all over the place. Synchronous processing can kill your bot if things go viral. Using Redis for async queuing and state was the only way to stay alive during traffic storms.
  • Trust & Safety is nuanced: Moderation isn’t just about weeding out obvious malware. It’s about protecting the vibe of the community. The most valuable thing an AI tool like this can do is know when to step back and let a human decide.

What's next for TriageOps

  • Multimodal Analysis: We’re working on expanding the ingestion engine to handle images, scrape text from memes, and scan outbound URLs for clickbait or malware. Not just straight .selftext.
  • Polymorphic Comment Integration: Comments are a whole separate beast. We’re adapting micro-batching to handle that high-speed stream, which means building a new type-checking layer since comment data isn’t laid out like submissions.
  • Subreddit-Specific Fine-Tuning: Every time a mod overrides an AI decision in the dashboard, I log that. Our next step is feeding those decisions back as training data, so the model keeps learning a subreddit’s inside jokes and unwritten rules. More local brain, less generic AI confusion.

Built With

Share this project:

Updates