Comment Triage Bot

Turn hours of YouTube comment chaos into a 5-minute review with structured LLM classification, priority scoring, and authentic style-matched drafting.

Comment

💡 Inspiration

Content creators with even a modest audience receive 50 to 500+ comments per video. Buried inside those comment threads are golden opportunities: high-value sponsorship inquiries and genuine community questions that drive engagement. But creators also face malicious crypto scams, telegram/whatsapp phishing bots, and toxic insults.

Today, creators face a lose-lose choice: spend hours manually scrolling and replying, or ignore comments entirely and hurt audience retention. We built Comment Triage Bot to transform a multi-hour manual chore into a five-minute focused review session.

⚙️ What It Does

Comment Triage Bot is an intelligent assistant built specifically for the creator workflow:

  1. Ingestion: Ingests real video comments using the YouTube Data API v3 (with built-in curated demo fallbacks for zero-setup evaluation).
  2. Batch Classification: Categorizes comments into 6 distinct buckets: question, sponsor_pitch, spam, toxic, positive, and negative with AI confidence scores and clear justifications.
  3. Priority Scoring: Ranks comments by algorithmic leverage so creators attend to urgent business inquiries and top-voted community questions first.
  4. Style-Matched Reply Drafting: Generates ready-to-send replies seeded with the creator's voice persona (Friendly & Concise, High Energy Hype, Deep Tech, or Business Executive) and past reply examples.
  5. Creator Studio Dashboard: A dark-mode workspace featuring category tabs, KPI metrics, real-time search/sort, inline draft editing, one-click clipboard copying, and CSV/JSON export.

🛠️ How We Built It

We designed a 3-layer architecture prioritizing speed, cost efficiency, and authentic creator tone:

  • Layer 1 — Data Ingestion: Built with FastAPI and the YouTube Data API v3 (videos and commentThreads.list endpoints). It extracts top-level comments, like counts, timestamps, and author avatars.
  • Layer 2 — Structured Batch Classification: To avoid slow and expensive per-comment API calls, comments are processed in batches of 15–20. We feed structured system prompts to LLMs (supporting OpenAI gpt-4o-mini, Anthropic claude-3-5-haiku, or an offline heuristic engine) enforcing strict JSON schemas.
  • Priority Leverage Scoring: We implemented an algorithmic ranking formula combining engagement leverage, recency decay, and commercial urgency: $$\text{Priority Score} = \left[6.5 \cdot \ln(1 + \text{likes}) + 4.0 \cdot \ln(1 + \text{replies})\right] + \text{Recency Decay Bonus} + \text{Category Weight}$$
  • Layer 3 — Style-Matched Reply Drafting: Comments marked needs_reply = true trigger a second prompt seeded with few-shot examples of the creator's real past replies. This ensures replies sound like the creator rather than generic, robotic AI text.
  • Frontend Dashboard: Built with modern HTML5, Tailwind CSS, Lucide icons, and vanilla JavaScript for zero-build-step responsiveness.

🧗 Challenges We Faced

  1. Minimizing Latency & API Costs: Processing 50–100 comments with individual LLM calls causes high latency and token bloat. We solved this by developing batching logic (15–20 comments per request) that reduces token usage by over 80% while retaining high classification accuracy.
  2. Avoiding Robotic AI Tone: Creators hate sounding like generic bots. We overcame this by implementing few-shot style seeding that mirrors the creator's actual phrasing, length, and emoji preferences.
  3. Zero-Setup Reliability: Requiring judges or users to set up Google Cloud projects and paid API keys creates friction. We added curated, authentic real-world video datasets (AI Agent Tutorial and M4 MacBook Pro Review) and an offline classification fallback so the entire pipeline runs reliably out of the box.

🏆 Accomplishments That We're Proud Of

  • A complete, functional text-in, text-out pipeline with zero video/rendering failure points.
  • Real-time priority ranking that bubbles lucrative brand sponsorships and high-impact questions directly to the top.
  • One-click copy with instant visual feedback and inline response editing.
  • Full multi-provider flexibility: works out of the box with YouTube Data API v3, OpenAI, Claude, or local offline fallbacks.

📚 What We Learned

  • How to design structured batch prompts with JSON schema validation for predictable multi-item LLM outputs.
  • The mathematical balance of logarithmic engagement scaling vs. exponential recency decay in content moderation.
  • How to craft few-shot prompts that genuinely match a creator's personal voice and vocabulary.

🔮 What's Next for Comment Triage Bot

  • OAuth 2.0 Live Publishing: Enable one-click reply publishing directly back to YouTube via OAuth.
  • Sentiment Shift Tracking: Interactive graphs tracking audience sentiment over the 7 days following a video drop.
  • Viral Comment Alerts: Early notification trigger when a comment experiences an unexpected spike in likes or sub-threads.

Built With

Share this project:

Updates

Submission history