ModSentinel

ModSentinel is a fast, AI-driven content moderation console designed to intercept orchestrated bot raids, financial scams, and targeted toxicity across multiple Reddit communities in real-time.

Comment

Inspiration

Reddit communities are under siege. Coordinated bot networks can deploy thousands of AI-generated financial scams and toxic harassment campaigns across subreddits in mere seconds—leaving volunteer moderators weaponless against machine-speed attacks. When a raid hits, moderators are completely overwhelmed, forced to manually lock threads and ban accounts one by one.

We realized that traditional keyword filters are too slow, brittle, and rigid to catch dynamic, AI-generated spam. We built ModSentinel to give Reddit moderators an "Enterprise-Grade Mission Control"—a system combining state-of-the-art LLMs with hyper-fast local heuristics to automatically quarantine threats before they ruin the community experience.

What it does

ModSentinel acts as an intelligent shield for Reddit communities. It features:

  • Live Threat Feed: A high-performance, unified dashboard that monitors real-time comment streams across single or multiple subreddits simultaneously.
  • Gemini AI Moderation Engine: Seamlessly integrates with the gemini-3.1-flash-lite model to analyze the semantic context of comments. It breaks down threats into specific vectors (Toxicity, Commercial Spam, Identity Attacks), moving far beyond rigid keyword matching.
  • Heuristic Bot Network Detection: A local, fast-pass memory state that tracks identical comment hashes across different authors and subreddits. It instantly quarantines crossposting bot swarms at zero API cost.
  • Dynamic Quarantine Filters: A toggleable system that automatically hides flagged comments from the community view, replacing them with a sleek, native-feeling "Removed by ModSentinel" badge.
  • API Telemetry Inspector: A developer-focused slide-out panel providing absolute transparency into the AI's decision-making process, showcasing real-time latency, token costs, and raw JSON payloads from the Gemini API.

How we built it

  • Architecture: We designed a hybrid, two-tier processing pipeline. Fast, deterministic heuristic checks run locally in the browser to catch obvious bot swarms instantly. Nuanced, context-heavy analysis is routed to the live Gemini API, maximizing both accuracy and cost efficiency.
  • AI Integration: We integrated directly with Google's Generative Language REST API (v1beta), specifically optimizing for the highly efficient gemini-3.1-flash-lite model. We utilized strict JSON schema enforcement via system instructions to guarantee deterministic outputs.
  • Frontend: We built a custom, dependency-free Virtual DOM implementation using Vanilla JavaScript to ensure minimal execution overhead, allowing the UI to render hundreds of incoming comments per minute without dropping frames.
  • Styling: We crafted a stunning, dark-mode-first aesthetic heavily inspired by modern cyber-security enterprise consoles and Reddit's native design language, using pure CSS variables and hardware-accelerated micro-animations.

Challenges we ran into

  • The LLM Safety Filter Paradox: The biggest technical hurdle was that Gemini's default safety filters actively blocked our moderation prompts. Because our test payloads contained simulated hate speech and aggressive scams from bot raids, the API initially refused to return classification data. We resolved this by diving into the API documentation and explicitly injecting safetySettings with BLOCK_NONE thresholds, forcing the model to act as an objective, clinical classifier rather than an easily offended end-user filter.
  • High-Velocity Performance Bottlenecks: Rendering a live feed of hundreds of DOM elements per minute caused significant layout thrashing and lag in early prototypes. We overcame this by implementing a custom virtual DOM wrapper (dom.js) and optimizing CSS repaints to achieve a buttery-smooth 60fps scrolling experience under heavy simulated load.
  • Distributed State Management: Tracking bot networks across multiple separate subreddits required engineering a robust, ephemeral crosspost cache that accurately hashes comment bodies and compares them against high-frequency historic intervals.

Accomplishments that we're proud of

  • The Enterprise Aesthetic: The UI looks and feels like a professional, high-tier security product. The animations are crisp, the dark mode is balanced, and the visual feedback is deeply satisfying.
  • The Hybrid Engine Routing: We are incredibly proud of the routing logic that successfully balances zero-cost local heuristics for obvious bot spam with the deep semantic understanding of Gemini 3.1 Flash Lite for nuanced harassment.
  • Live Telemetry: The sliding "API Inspector" panel is a fantastic engineering feature that completely demystifies the AI, proving to users exactly how the model evaluates threats in real-time.

What we learned

  • API Nuances: We gained deep experience working directly with LLM REST APIs, specifically handling model versioning, strict schema enforcement, and navigating safety overrides.
  • DOM Optimization: We developed a deep appreciation for Vanilla JS performance tuning, learning firsthand how to mitigate the actual performance cost of DOM reflows when handling high-velocity data streams.

What's next for ModSentinel

  • Reddit Native API & Devvit Integration: The next step is connecting ModSentinel directly to the official Reddit API via OAuth and expanding it within the Devvit ecosystem to run natively on live subreddits.
  • Automated Actioning Rule-Engine: Implementing customizable automation rules (e.g., "If Toxicity Score > 0.9 and Confidence > 95%, automatically issue a 3-day ban").
  • Multi-Modal Analysis: Expanding the Gemini integration to analyze linked images, memes, and videos for automated image safety and embedded scam text.

Built With

Share this project:

Updates