Retention Autopsy

Stop guessing why viewers click away. Mathematically diagnose retention cliffs, cross-reference transcripts with AI, and compile a custom playbook for your channel

Comment

💡 Inspiration

Every YouTube creator has stared at their Audience Retention graph in YouTube Studio, trying to figure out why 15% of their audience suddenly dipped at exactly 4 minutes and 12 seconds. Was it a boring tangent? A poorly placed sponsor read? Bad pacing? Or just normal drop-off?

Currently, YouTube Analytics only gives you a static chart. To find the root cause, you have to manually open your editing timeline, match up the timestamp, and guess. We wanted to build a tool that takes the guesswork out of retention analysis. Retention Autopsy automates this entire diagnostic process and mines those insights into a channel-specific "memory" that scores scripts before you even record.

⚙️ What it Does

Retention Autopsy is a diagnostic pipeline and dashboard for YouTube creators:

  1. Precision Cliff Detection: It fetches your 100-bucket audience retention curve and applies signal processing math to isolate real retention drops (cliffs) and gains (spikes) from natural, expected viewer decay.
  2. Transcript Integration: It maps the timestamp ranges of these cliffs directly to the corresponding text segments in your video's transcript.
  3. Automated Labeling (Heuristics & AI): It processes the context around the cliff to label why it happened (e.g., sponsor_read, intro_ramble, subscribe_beg, dead_air, chapter_boundary) using deterministic rules first, falling back to Anthropic's Claude LLM with a strict vocabulary parser.
  4. Channel Memory: It aggregates findings across your entire video catalog. If a pattern repeats (e.g., "Sponsor reads placed before the 2-minute mark cause an average 15% drop"), it adds it to your channel-specific rulebook to help prevent future mistakes.

🛠️ How We Built It

We engineered Retention Autopsy to run as a single-deploy Next.js web application with a lightweight, high-performance architecture.

Tech Stack

  • Framework: Next.js 16 (App Router, TypeScript)
  • Database: Supabase Postgres
  • Mathematical Processing: Client-side and Edge-safe array math (Z-score analysis)
  • LLM Engine: Anthropic SDK (Claude 3.5 Sonnet)
  • Animations & Styling: Tailwind CSS v4, Framer Motion, and Lenis Smooth Scroll
  • Testing: Vitest for test-driven pipeline development

The Pipeline Architecture

Our data pipeline consists of 4 distinct, testable steps:

  1. fetchChannelData: Gathers video metadata via the YouTube Data API v3, extracts retention curves via the YouTube Analytics API, and pulls closed captions. (Features a fallback to manual .srt uploads when the API 403s on auto-generated tracks).
  2. detectCliffs:
    • Smooths the 100-bucket curve using a moving average window of 3.
    • Fits a custom decay baseline curve unique to the video's length and format.
    • Computes Z-scores: $z[i] = \text{residual} / \text{stdev(residuals)}$.
    • Flags data points where $|z| > 2.0$ AND the magnitude change $|\Delta| > 3$ percentage points.
    • Merges adjacent flagged buckets into single, discrete time windows.
  3. labelCliffs: Uses pattern matching for common keywords (sponsor name, "subscribe", long silences) to assign labels. If confidence is low ($<0.6$), the segment is analyzed by Claude with a strict vocabulary parser.
  4. minePatterns: Aggregates all labeled cliffs across the channel to calculate the average impact (retention delta) and records patterns appearing 3 or more times ($n \geq 3$).

Premium UI/UX Redesign

We completely overhauled the user experience to match modern design standards:

  • Glassmorphism: Frosted translucent cards (GlassCard) utilizing backdrop-blur-md, subtle border highlights (border-white/20), and gradient shadows.
  • Animated Elements: Flowing radial gradient blobs moving dynamically in the background using CSS keyframe animations.
  • Smooth Scrolling: Integrated Lenis Scroll to enhance the feel of page navigations and parallax movements.
  • Typography: Loaded the high-character Google Font Edu VIC WA NT Hand globally to give the dashboard a polished, handwritten "autopsy log" aesthetic.

🚧 Challenges We Ran Into

  • Timestamp Resolution Fuzziness: YouTube Analytics returns retention data in 100 buckets, meaning a 10-minute video has a 6-second resolution, while a 1-hour video has 36-second resolution. Representing this as precise timestamps would be misleading. We solved this by structuring our database using Postgres integer ranges (int4range) to properly preserve the fuzzy nature of the time windows.
  • YouTube Captions API Restrictions: The API restricts ASR (auto-generated) caption downloads for third-party OAuth apps. We engineered a seamless fallback upload drag-and-drop zone that accepts .srt and .vtt formats so creators are never locked out.
  • Data Scarcity (Cold Start): Running complex machine learning on 30-40 videos per channel leads to overfitting. Instead of faking analytics, we built strict honesty rules: if a channel has $<3$ videos analyzed, the UI displays "insufficient history" rather than generating low-confidence patterns.

🏆 Accomplishments That We're Proud Of

  • Noise-Resilient Math: Our cliff detection algorithm ignores normal viewer decay (like the steady drop-off typical of all YouTube videos) and isolates actual editor mistakes.
  • Unified Pipeline: We achieved complex mathematical analytics and transcript matching directly within Next.js API routes without needing separate Celery/Redis worker queues.
  • Visually Striking UX: The dashboard feels cohesive, interactive, and premium, avoiding plain standard layouts in favor of animated backgrounds and glass components.

📚 What We Learned

  • How to normalize and smooth raw analytics arrays to find outliers (Z-scores).
  • Designing database structures tailored to interval ranges instead of absolute time points.
  • How to strictly guide and parse LLM JSON responses to ensure they perfectly map back to static relational database schemas.

🚀 What's Next for Retention Autopsy

  • Shorts Analytics: Adapting the baseline math for video formats under 90 seconds, where retention curves often exceed 100% due to loop views.
  • Script Pre-Auditor: A browser extension (Chrome/Google Docs) that analyzes your video script before you film, checking it against your channel memory to warn you of potential drops (e.g., "Your intro is 45 seconds long; your channel memory predicts a 12% drop here").

Built With

Share this project:

Updates