ModTrials

• 

Inspiration

Moderation teams are under pressure to move fast: reduce spam, protect community quality, and keep the queue manageable. But every stricter rule has a hidden cost. A rule that catches obvious spam can also catch good-faith posts, new users, researchers, builders, or members who simply used the wrong wording.

Most moderation tools focus on action: remove, warn, ban, approve, or vote. ModTrials focuses on the missing step before action:

Can this rule be trusted before it affects users?

Most mod tools help teams act on content already in the queue. ModTrials helps teams test rules before those rules create queue load, false positives, or user-facing mistakes.

We built ModTrials as a staging environment for subreddit rules. Instead of moving a new rule directly into enforcement, moderators can shadow-test it on real community activity, measure false positives and gray-area cases, and launch only when there is evidence.

What it does

ModTrials lets subreddit moderators run private shadow trials for moderation rules on real Reddit posts and comments.

A moderator can start a trial from a native Reddit post/comment menu or with a bot command:

u/modtrials trial this --dm

ModTrials privately DMs the moderator a counterfactual result:

"If this rule were live, this post would have been held for review. No user-facing action was taken."

The app records why the item matched, stores privacy-minimized evidence, and lets moderators label examples as good catches, false positives, gray-area cases, or rules that need rewriting.

The dashboard then turns that evidence into practical launch signals:

• readiness score
• false-positive risk
• gray-area risk
• confidence level
• queue-load signal
• recommended launch mode: warning, repair-first, hold-for-review, do not auto-remove, or rewrite

The latest version also includes a rule flight recorder, rule comparison, counterfactual trial results, and privacy receipts so moderators can see how each rule is evolving before launch.

ModTrials does not auto-ban, auto-remove, use AI, train models, or publicly accuse users.

How we built it

We built ModTrials as a Reddit Devvit app with a React dashboard, Hono server, Devvit Redis, Devvit Reddit API, menu actions, triggers, scheduler tasks, private messages, and custom posts.

The primary UX is native to Reddit. Moderators can run trials from post/comment menus or bot-style commands without leaving the subreddit. The dashboard is a secondary audit and readiness surface for reviewing rule evidence.

The rule engine is deterministic. It evaluates conditions such as keywords, external links, domains, short text, flair, and account-age style checks. Trial events are stored in Devvit Redis with privacy minimization: no usernames, no full post/comment body, no raw title, and no raw URL. Redis keys are scoped by subreddit so one community cannot read or mutate another community's ModTrials rules or evidence.

We also added server-side moderator permission checks for all menu actions and dashboard API endpoints. If a non-moderator opens a dashboard post, they see a locked state and no moderation data is returned.

Challenges we ran into

The hardest product challenge was resisting the temptation to build another enforcement bot. A tool that automatically removes content may look powerful, but it can also create false positives, user frustration, and more work for moderators.

So we designed ModTrials to be non-destructive by default. It creates evidence, not punishments.

We also had to handle real Devvit runtime behavior. Reddit can rewrite user mentions into markdown links, comment submit/create triggers can both fire for the same command, and delete triggers provide different payload shapes from normal comment events. We added command de-duplication, markdown mention normalization, and deletion cleanup to make the workflow reliable.

Privacy was another major challenge. Moderators need useful evidence, but the app should not become a user-profiling database. ModTrials minimizes trial events before writing to Redis and includes a privacy receipt in private results.

Accomplishments that we're proud of

We built a real end-to-end Devvit moderation app, not a mock demo.

ModTrials can be installed on a subreddit, read real posts/comments, start private trials from native Reddit actions, process bot commands, send private DM results, remove command comments after processing, store minimized evidence in Redis, label outcomes, compare rule safety, and compute launch readiness.

We are proud that ModTrials occupies a clear product lane:

Other mod tools help teams act on content. ModTrials helps teams prove a rule is safe before it acts on users.

That makes it useful alongside Automod-style tools, queue tools, and collaborative review tools instead of competing with them directly.

What we learned

We learned that moderator trust depends on restraint. The best tool is not always the one that takes the fastest action. Sometimes the highest-value tool is the one that prevents the wrong action.

We also learned that privacy and UX are connected. A moderation tool feels more trustworthy when it works inline, sends private results, avoids public accusations, and stores only the minimum evidence needed.

On the technical side, we learned how to work with Devvit menu actions, triggers, Redis storage, custom posts, private messages, moderator permission checks, and public review requirements.

What's next for ModTrials

Next, we want to make ModTrials a fuller rule safety lifecycle:

• stronger rule flight recorder reports
• side-by-side rule version comparison
• team-level review workflows
• repair-first launch recommendations
• weekly private rule health summaries
• safer handoff from trial mode to Automod-style enforcement

Longer term, ModTrials can become the staging layer for subreddit governance changes: a place where moderation teams test policy before it becomes punishment.

Built With

• hono
• node.js
• react
• redis
• typescript
• vite
• vitest
• zod