Lighthouse

A Devvit mod tool that flags likely AI-written content, explains why, and gives moderators a fast review queue with feedback, notes, and calibration.

Comment

Inspiration

A lot of Reddit moderation work now happens in a gray area. A post may look AI-written, but removing it blindly is unfair. Ignoring it can also make high-effort communities harder to moderate.

We wanted Lighthouse to sit in that middle space: not an auto-ban bot, but a review tool that gives moderators a clear signal, the evidence behind it, and a fast way to make the final call.

What it does

Lighthouse reviews new posts and comments and flags content that looks likely to be AI-generated. Each flagged item gets a confidence score, a short explanation, and an evidence panel.

Moderators can:

  • Review flagged posts and comments from a custom Devvit dashboard
  • See local linguistic signals behind the score
  • Use an optional LLM second pass for borderline cases
  • Mark items as AI, human, or unsure
  • Add mod notes
  • Apply a removal reason
  • Track recent detections, repeat offenders, calibration, and activity
  • Configure thresholds, budgets, provider settings, and automation from the app

The goal is to save moderator time while still leaving judgment with the mod team.

How we built it

Lighthouse is built as a Reddit Devvit app with a React custom post interface.

The detection pipeline has two parts. First, a local classifier extracts writing signals such as paragraph regularity, phrase patterns, sentence variation, punctuation usage, and personal-marker density. If the score lands in the gray zone, Lighthouse can send a redacted version of the text to an LLM for a second review.

The app stores verdicts, feedback, dashboard metrics, cache entries, audit activity, and calibration data in Devvit Redis. The dashboard and review cards are rendered as Devvit web views so moderators can use the tool directly inside Reddit.

Challenges we ran into

The hardest part was making the tool useful without making it feel too confident. AI detection is not perfect, so the product had to be designed around review, evidence, and feedback instead of automatic enforcement.

Another challenge was fitting a real moderation workflow into Reddit’s mobile UI. Custom posts have limited space, and the Reddit app has its own scrolling behavior, so a lot of time went into making the dashboard and review cards compact enough to use on mobile.

The LLM layer also needed guardrails. We added redaction, JSON validation, caching, retry handling, circuit breakers, and daily budget caps so communities are not surprised by cost or unreliable provider behavior.

Accomplishments that we're proud of

We’re proud that Lighthouse is more than a score. It has a working dashboard, queue, evidence panel, settings, feedback actions, mod-note flow, removal-reason support, audit log, weekly digest, and calibration recommendations.

We’re also proud that the app works even without an LLM key. Communities can start with the local detector, then turn on LLM review later if they want more coverage for borderline cases.

What we learned

We learned that moderator tools need to be careful about trust. A tool can be technically impressive and still be bad for mods if it hides too much or acts too aggressively.

The best version of Lighthouse is not “the bot knows best.” It is “the bot did the first pass, here is why, and here is the fastest way for a human mod to decide.”

What's next for Lighthouse

Next, we want to improve calibration with more real moderator feedback, add better subreddit-specific tuning, and make the dashboard easier to scan on very busy communities.

We also want to add richer team workflows, such as assigning review items, showing disagreement between moderators, and summarizing patterns across users or threads.

Built With

Share this project:

Updates