Moment Matcher

Inspiration

Most “contextual” ad tech still looks at keywords, not moments. But video is narrative, tone, and timing. We were inspired to build a system that understands Moments in video and places ads at the exact second viewers are receptive—protecting brand safety while lifting revenue.

What it does

Moment Matcher analyzes a video, finds optimal ad insertion seconds, and recommends a slate that maximizes Engagement, Brand–Content Alignment, and Revenue under publisher constraints.

Builds Seeds from chapters/highlights/search.
Creates Moments (Pre/Now/Post) around each Seed.
Runs a 200-audience LLM simulator that outputs scores and an action distribution per (Audience, Seed, Ad):
watch_full, skip_after_5s, skip_immediately, cta, ignore, pause_to_research.
Predicts CTR with confidence bands (quantile regression), computes BrandMatch, and projects Revenue using historical eCPM & retention.
Selects placements via Voting → Greedy, honoring min-gap, per-minute caps, and safety.

How we built it

TwelveLabs API for Chapters/Highlights/Search to create Seeds and anchor points.
Gemini to generate compact Moments context and to score audience responses (strict JSON).
Brand–content alignment via TwelveLabs text-to-video search + embedding similarity, combined with LLM tone and safety.
Action-based E2E scoring: we convert LLM action probabilities into engagement/revenue:
- CTR_hat = P(cta)
- Retention after the ad adjusts with P(skip_immediately) and P(skip_after_5s).
Quantile regression (p10/p50/p90) with calibration for CTR; revenue = eCPM × impressions × CTR_hat / 1000.
Dashboard: What-If controls re-optimize without re-calling the LLM for sub-second UX.

Challenges we ran into

Consistent JSON from LLMs: solved with strict schemas, retries, and validators.
Latency & cost at scale: two-stage evaluation (light filter → batched LLM), caching by (audience, ad, micro_bucket).
Safety & tone edge cases: no-go windows + proximity penalties with human-readable rationales.
Placement stability: keyframe/silence snapping and small local search to avoid jarring cuts.

Accomplishments that we're proud of

An end-to-end system that turns Moments into monetizable slots with confidence bands—trustworthy by design.
200-audience simulator that outputs action distributions, enabling realistic engagement/retention effects instead of single scores.

What we learned

Transitions matter: ads land best right after a narrative beat resolves; worst just before highlights.
Ensembles win: LLM reasoning + structural/audio/visual features outperform either alone.
Show uncertainty: p10–p90 bands prevent over-promising and improve decision quality.
UX is critical: real-time re-optimization (no extra LLM calls) turns a demo into an operational tool.

What's next for Moment Matcher

Online learning from real CTR/skip outcomes; feedback loops per audience/creative.
Creative-aware scoring: thumbnails/captions variants and creative–moment matching.
Campaign pacing & rotation across episodes/playlists; multi-ad slate optimization.
Deeper safety packs and brand-specific policy templates; A/B experimentation APIs.
Partner integrations (ad servers/exchanges) for push-button activation.