Ask the Video

Watch any YouTube video, hit one button, and ask the creator anything. Powered by AI.

Comment

Ask the Video — Project Story

💡 Inspiration

It started with a frustration I've had a hundred times.

You're watching a YouTube tutorial at 1am. The creator explains something, and you almost get it — but not quite. You have one specific question. You pause. You scroll through 47,000 comments. You type your question. You wait. Nothing.

The creator has 2 million subscribers. They're not coming.

I kept thinking — what if the video itself could answer back? Not a chatbot. Not a FAQ. The actual creator — their knowledge, their tone, their way of explaining things — reconstructed and available the exact moment you need them, without leaving the page.

That frustration is what became Ask the Video.

🔨 How I Built It

The core insight is simple: every YouTube video is already a knowledge base. The transcript contains everything the creator knows about that topic. The challenge is making that knowledge feel human — and delivering it without breaking the viewer's flow.

Why a Browser Extension (Not a Web App)

The decision to build a Chrome extension was deliberate:

Factor Extension Web App
Context Lives on YouTube — reads video ID, timestamp, transcript automatically Requires copy-pasting links
Voice input Captures voice while video is paused in the same tab Requires switching tabs
Friction Near-zero High

Pause → Click → Ask → Answer. That's the entire UX. A web app would have broken it.

The Architecture

YouTube Page (Content Script)
    ↓ video ID, timestamp, transcript
Background Service Worker
    ↓ question + context
Backend API
    ├── Transcript Service
    ├── LLM Engine (Creator Persona)
    └── TTS Engine (Voice Reply)

Step 1 — Content Script Injection

The extension injects a content script into every youtube.com/watch page. It listens for YouTube's player state events — specifically the pause event — and surfaces a floating button only when the video is paused. When the user resumes, the button disappears. Non-intrusive by design.

Step 2 — Context Extraction

When the user opens the widget, three things are captured automatically:

  • Video ID from the URL (?v=VIDEO_ID)
  • Current timestamp from the player's currentTime
  • Transcript from YouTube's caption data if available

Step 3 — Persona Construction

The transcript is fed to the LLM with a prompt that extracts the creator's tone, vocabulary, and explanation style. This is cached per video so repeat questions are instant.

The system prompt that makes the persona work:

You are [Creator Name]. You speak exactly like them: [style traits].
You are answering a viewer's question about your video titled '[Video Title]'.
The viewer paused at timestamp [MM:SS]. Here is the relevant transcript:
[Transcript chunk]

Answer in the first person, as the creator, directly to the viewer.
Stay in character. If irrelevant to the video, return: IRRELEVANT.

Step 4 — Voice Pipeline

  • Input: Web Speech API captures the user's question (hold-to-speak)
  • Output: TTS converts the answer to audio that auto-plays in the widget

Why This Works — The Relevance Gate

Every question passes through a relevance filter before the persona responds. Conceptually, if we think of the question $q$ and transcript $T$ as vectors in semantic space, a question is relevant if:

$$\text{similarity}(q, T) = \frac{q \cdot T}{|q| \cdot |T|} > \theta$$

In practice, we let the LLM make this judgment rather than computing embeddings explicitly — it handles nuanced edge cases better and is faster at this scale.

📚 What I Learned

Prompt engineering is product design. The difference between "answers from the transcript" and "answers as the creator" is one line in the system prompt — but the felt experience is completely different. Persona is a UX decision, not just a technical one.

The extension context is everything. Having the video ID, timestamp, and transcript automatically — without asking the user for anything — is what makes the experience feel magical. If the user had to do any of that manually, the product would die.

Voice changes the product category. Adding hold-to-speak voice input didn't just add a feature — it shifted the product from "AI chatbot on YouTube" to "talk to the creator." That's a different product.

Ship the core loop first. The temptation was to build creator voice cloning, multi-video personas, history sync. None of that shipped. What shipped was: pause → ask → answer. That loop working well is worth more than ten half-built features.

🚧 Challenges

1. Content script fragility. YouTube's DOM changes frequently. Selectors that work today break on the next deploy. The fix was using MutationObserver to watch for player state changes rather than relying on static DOM queries.

2. Manifest V3 microphone access. Chrome's Manifest V3 doesn't allow microphone access directly from content scripts. Audio capture has to go through an offscreen document — an extra layer of message passing that added complexity to the voice pipeline.

3. Transcript quality varies wildly. Auto-generated captions have no punctuation, inconsistent formatting, and occasional gibberish. Getting the LLM to extract a coherent persona from raw caption data required careful preprocessing and prompt design.

4. Staying in character without hallucinating. The creator persona needs to feel natural — but it can't invent facts the video never mentioned. The balance between sounds like them and stays accurate was the hardest prompt to tune. The solution was grounding answers strictly in the transcript segment around the current timestamp, with common sense extensions only for directly implied knowledge.

5. Relevance gate calibration. Early versions blocked too aggressively — even reasonable adjacent questions got flagged. Late versions were too permissive. The sweet spot was prompting the model to apply generous common sense before declaring something irrelevant, and only hard-blocking questions with zero connection to the video topic.

🚀 What's Next

  • Creator voice cloning — TTS that actually sounds like the creator using ElevenLabs or similar
  • Multi-video persona — build the creator twin from their entire channel, not just one video
  • Timestamp-linked answers — answers that link back to the exact moment in the video they reference
  • Creator dashboard — let creators claim their AI persona, review questions, and customise their twin
  • Firefox + Edge — the architecture is Manifest V3 compatible, ports are straightforward

Built With

Share this project:

Updates

Submission history