YouTube ChatBot (VideoMind)

Turn any YouTube video into an interactive AI knowledge source—ask questions, find exact moments, and get context-aware answers instantly.

Comment

Project Story: YouTube ChatBot (VideoMind)

Inspiration

We’ve all been there: sitting through a 45-minute technical tutorial or a 2-hour podcast, trying to find that one specific explanation or code snippet mentioned somewhere in the middle. Scrubbing through the timeline, blindly clicking, and hoping to hit the right frame is frustrating and time-consuming.

We built YouTube ChatBot (VideoMind) to eliminate the friction of video-based learning and research. We wanted an assistant that acts like a real-time copilot while you watch YouTube: an extension that doesn't just skim surface-level captions, but deeply indexes video transcripts, grounds every claim with verified timestamps, generates interactive study materials, and allows instant navigation—all without forcing you to leave the video player.

What it does

VideoMind is a full-stack AI copilot delivered as a native Chrome Extension backed by a high-performance local FastAPI engine. When you open any YouTube video, VideoMind auto-detects it, retrieves and cleans the transcript, builds a vector index, and provides an interactive floating chat interface on the bottom-left of your screen.

Key capabilities include:

  • Grounded Q&A with Evidence Timestamps: Ask any question about the video. Every response includes evidence citations with ▶ Jump to buttons that sync directly with the YouTube player to take you to the exact second where the topic is discussed.
  • Proactive Summaries & Instant Study Tools: Automatically generates structured video summaries, key takeaways, topic breakdowns, auto-generated quizzes (MCQ, True/False, Short Answer), interactive flashcards, and exportable Markdown/PDF study notes.
  • Multi-Language & Quality Repair: Automatically ingests transcripts in English, Hindi, and other languages, using an LLM quality repair pass to fix mangled or unpunctuated auto-captions.
  • "Ask This Moment" (Spatial-Temporal RAG): Evaluates your current video playback position and extracts a temporal window ($\pm 80\text{s} / \pm 50\text{s}$) to answer queries specifically about what's currently happening on screen.
  • Side-by-Side Video Comparison: Compares two videos on the same topic, generating a comparative analysis matrix and a "Who should watch which" recommendation.
  • External Web & Code Research: Automatically triggers web searches (via Tavily/DuckDuckGo) or code lookups (via GitHub and Stack Overflow) when queries require up-to-date or deeply technical verification outside the transcript.

How we built it

┌─────────────────────────┐         HTTP / JSON         ┌─────────────────────────────────┐
│    Chrome Extension     │ ──────────────────────────> │         FastAPI Backend         │
│  (Manifest v3 / JS)     │ <────────────────────────── │    (app.py / extension_app.py)   │
└─────────────────────────┘                             └─────────────────────────────────┘
                                                                         │
                                                                         ▼
                                                        ┌─────────────────────────────────┐
                                                        │          RAG Pipeline           │
                                                        │  (FAISS + LangChain Indexing)   │
                                                        └─────────────────────────────────┘
                                                                         │
                                                                         ▼
                                                        ┌─────────────────────────────────┐
                                                        │    Cascading LLM / Embeddings   │
                                                        │ Gemini ➔ OpenRouter ➔ Ollama    │
                                                        └─────────────────────────────────┘

The system architecture is separated into three core layers:

  1. Chrome Extension UI (Manifest V3): Built using modular Vanilla JavaScript (IIFE) and CSS custom properties. It runs non-intrusively as an injected overlay, monitoring URL changes, controlling video playback through the DOM, and parsing structured Evidence: blocks from the LLM to render interactive jump links.
  2. FastAPI Backend (15+ Endpoints): Acts as the orchestration engine handling non-streaming/streaming Q&A (/ask, /ask-stream), lightweight off-topic checking (/ask-check), vector pre-warming (/prepare-video), comparative analysis (/compare-videos), and ReportLab-powered PDF generation (/transcript-pdf, /notes-pdf).
  3. Multi-Stage RAG Pipeline:
  4. Ingestion & Repair: Fetches transcripts via youtube-transcript-api and runs quality scoring to fix low-punctuation captions.
  5. Chunking & Vector Store: Splits text into semantic chunks while maintaining metadata mapping $T_{\text{start}}$ and $T_{\text{end}}$. Embeddings are stored in a FAISS vector index using a cascading provider model:

$$\text{Embeddings: } \text{SentenceTransformer} \longrightarrow \text{Gemini} \longrightarrow \text{Ollama}$$

  • Retrieval & LLM Cascade: Performs top-$k$ similarity search ($k=5$) with optional multi-query expansion. Standard generation cascades gracefully to maintain 100% uptime:

$$\text{LLM Chain: } \text{Gemini 2.5 Flash / Pro} \longrightarrow \text{OpenRouter (Free Tier)} \longrightarrow \text{Ollama (Local Qwen2.5:7b)}$$

Challenges we ran into

1. Noisy and Unpunctuated Auto-Captions

YouTube auto-generated captions often lack capitalization, punctuation, and line breaks, causing naive chunking strategies to split sentences across arbitrary boundaries. We solved this by developing a pre-embedding assessment step (Transcript_Processing.py) that measures caption quality and routes poorly formatted transcripts through an LLM repair pass prior to vector indexing.

2. Ensuring Exact Temporal Alignment

Vector search naturally matches semantics, but users expect accurate timestamp links. Keeping chunk start ($T_1$) and end ($T_2$) boundaries tied to their metadata during text splitting required careful indexing:

$$\text{Chunk}_i = \left( \text{text}_i, \text{meta} = { \text{start}: T_1, \text{end}: T_2 } \right)$$

We also engineered the system prompt in augmentation.py to strictly force the LLM to output evidence in a unified syntax, such as Evidence: [04:12 - 04:45], which the frontend extracts and turns into jump triggers.

3. Latency & LLM Rate Limits

Running a full RAG pipeline for every chat message introduced noticeable latency. We mitigated this by:

  • Implementing aggressive disk caching (.cache/) for FAISS indices, transcript JSONs, and 6-turn chat history.
  • Adding a lightweight /ask-check pre-check endpoint to detect off-topic questions in milliseconds before triggering an expensive vector search.
  • Supporting chunked SSE streaming (/ask-stream) so users see answers start streaming in under a second.

Accomplishments that we're proud of

  • Zero-Hallucination Evidence Layer: Successfully built a mechanism where clicking a cited source in an AI-generated answer immediately seeks the YouTube video to that precise moment.
  • 100% Uptime Architecture: Designed a 3-tier fallback strategy (Gemini $\rightarrow$ OpenRouter $\rightarrow$ Ollama) that ensures the system continues to operate seamlessly even if cloud APIs face outages or rate limits.
  • In-Extension PDF Exporting: Built custom backend export capabilities using ReportLab with Noto Sans Devanagari font rendering, allowing users to export full transcripts and study guides with full Hindi and Marathi language support.
  • Safety Guardrails: Integrated pattern-based pre-LLM safety filters (content_safety.py) to catch self-harm, weapons, CSAE, and malware queries before invoking remote LLMs, reducing costs and providing immediate crisis resource routing when needed.

What we learned

  • Prompt Engineering for Structured Outputs: Forcing LLMs to adhere to complex formats (such as citing evidence without hallucinating timestamps) requires strict system constraints, few-shot examples, and robust output cleaning algorithms.
  • Local RAG Efficiency: We discovered that local models like qwen2.5:7b paired with all-MiniLM-L6-v2 embeddings via Ollama can perform surprisingly well for grounded Q&A tasks when cloud access is unavailable.
  • Mathematical RAG Evaluation: Using the RAGAS framework (ragas_eval.py) taught us how to objectively score our retrieval and generation steps using metrics like Answer Faithfulness ($F$), Context Recall ($R$), and Answer Relevance ($A$).

$$\text{Faithfulness Score} = \frac{\vert{}\text{Verifiable Statements Grounded in Context}\vert{}}{\vert{}\text{Total Statements Generated}\vert{}}$$

What's next for YouTube ChatBot (VideoMind)

  • Multimodal Visual RAG: Moving beyond text transcripts by analyzing keyframes, slides, code on screen, and visual diagrams directly using vision-capable models (e.g., Gemini 2.5 Flash Vision).
  • Multi-Video Knowledge Graphs: Allowing users to create collections or playlists and query across hours of video content simultaneously.
  • Cross-Browser & Mobile Support: Packaging the extension for Firefox, Edge, and Safari, alongside a standalone web app for analyzing unlisted or local MP4 video files.

Built With

  • chrome
  • content-safety
  • css-custom-properties
  • duckduckgo-search
  • faiss
  • fastapi
  • gemini-api
  • javascript
  • langchain
  • manifest-v3
  • ollama
  • openrouter
  • pandas
  • pydantic
  • python
  • rag
  • ragas
  • reportlab
  • sentence-transformers
  • slowapi
  • tavily-search
  • uvicorn
  • vector-database
  • youtube-transcript-api
Share this project:

Updates

Submission history