Reeled In

💡 Inspiration

Short-form video is a game of a thousand tiny variables — the hook, the music, the cut rhythm, the on-screen text, the CTA — and creators have no cheap way to know which combination actually holds attention before they post. Today you either guess, post it live and hope, or burn ad spend to find out. Real testing means focus groups or eye-tracking studies that cost more than the video did.

Then Meta open-sourced TRIBE v2, a brain-encoding "digital twin" that predicts the fMRI response a human brain would have to a video — without scanning anyone. We realized you could use it as a pre-flight attention check: score an edit against a model of the brain and pick the winner before it ever goes out.

The name says the whole idea: an Instagram Reel + reeling you in. The metric is literally whether your content hooks and holds the viewer's brain.

🎣 What it does

You upload two or more versions of a Reel — or one base video plus a few voiceover scripts, and we generate the voice variants for you. reeled in predicts how the brain engages with each one, second by second, and declares a winner for the objective you care about (hold attention to the CTA, spike early, etc.) — all pre-publish, in seconds, no humans.

We're honest about what it is, on purpose: TRIBE predicts neural activation, not views or sales. It's a fast, directional proxy — a gut-check with a brain behind it, not a crystal ball.

🧠 How the scoring works

TRIBE predicts a whole-brain response over ~20,484 cortical vertices at 1 Hz. We reduce that to five interpretable functional networks — visual, auditory, language, motion, and default-mode (meaning/engagement) — and blend them into one composite engagement curve $E$:

$$E[t] = 0.30\,\mathrm{DMN}[t] + 0.25\,\mathrm{Vis}[t] + 0.20\,\mathrm{Lang}[t] + 0.15\,\mathrm{Aud}[t] + 0.10\,\mathrm{Mot}[t]$$

From $E$ over $N$ seconds we compute the four metrics a creator can optimize for:

$$\text{peak} = \max_{t} E[t], \qquad \text{sustained} = \frac{1}{N}\sum_{t=1}^{N} E[t]$$

$$\text{retention} = \frac{\operatorname{mean}!\big(E_{\text{final third}}\big)}{\operatorname{mean}!\big(E_{\text{first third}}\big)}, \qquad \text{overall} = 0.5\,\text{sustained} + 0.3\,\text{retention} + 0.2\,\text{peak}$$

The winner is simply the variant with the highest score on the objective you pick.

🛠️ How we built it

• Scoring engine (the core): TRIBE v2 on a Modal A100 GPU. It stands on three frozen encoders — V-JEPA 2 (video), Wav2Vec-BERT 2.0 (audio), LLaMA 3.2 (text) — fused by a transformer and projected onto the cortical surface, then reduced to the five networks via ICA.
• Backend: FastAPI on Modal orchestrates everything; MongoDB Atlas is the system of record.
• Frontend: built on Base44, with Auth0 for login.
• Generation & intelligence: ElevenLabs + ffmpeg generate and mux voice variants; Gemini watches the base video (multimodal) and proposes a test plan; Backboard provides LLM memory/RAG so recommendations personalize over time.
• How four of us moved fast: we froze a single CONTRACTS.md (the Score Object + the API) in hour one, then everyone built against mocks in parallel and snapped it together at the end — zero renaming, minimal blocking.

🧗 Challenges we ran into

• Our frontend and backend couldn't share a source of truth. On Base44's free plan we couldn't connect the project to GitHub — but our entire backend lived in a Git repo. So the Base44 frontend and our FastAPI backend evolved in two separate worlds, and we kept them in sync by hand against CONTRACTS.md instead of one shared codebase. Freezing that contract in hour one is the only reason it held together.
• An auth landmine, caught before demo night. We'd assumed the frontend could pass its login token to our backend — until we found that on the free tier, the platform's login can't be verified by an external server (no readable token; verification is paywalled). It would have silently broken frontend↔backend integration on stage. We caught it early and pivoted to a client-side Auth0 SPA flow our backend verifies against Auth0's JWKS.
• The winner was ranking backwards. Our first scoring pass normalized each clip to its own 0–1 range — and on a known boring-vs-engaging pair, it called the winner backwards. Stretching every clip to fill 0–1 flattens the differences between clips and saturates peak near 1.0, which is fatal when the whole product is comparing variants. We switched to a shared scale (all variants scored against one batch-wide reference), which got three blind pairs right. Since the Score Object is shared "law" across our four lanes, we documented the change and got team sign-off rather than editing the contract unilaterally.
• Making a whole brain legible to someone with zero neuroscience background. TRIBE hands us ~20,000 cortical values every second — dense, abstract, meaningless to a creator. The real design problem was deciding what to actually show: we distilled it to five named networks, one engagement curve, a per-second brain flipbook, and plain-English captions, so someone with no context can glance at it and immediately get "this edit hooks harder."
• Orchestrating four models with four different failure modes. A single run can touch a GPU brain model, a multimodal Gemini call, ElevenLabs TTS, and Backboard's memory — each with its own latency and ways to fail. We built the pipeline so any one stumbling degrades gracefully instead of breaking the screen (a Backboard timeout falls back to general tips), so the demo survives even if one service hiccups.
• Free-tier rate limits mid-build. Our Gemini key kept throwing 429s on larger video payloads right when we needed /suggest. We rerouted the same model through OpenRouter and kept moving.

📚 What we learned

• How brain-encoding foundation models actually work — frozen multimodal encoders (V-JEPA 2, Wav2Vec-BERT, LLaMA), a fusion transformer, cortical projection, and recovering known functional networks with ICA.
• A metric is only as good as its baseline. The model call is one line; the real work
  • and the real bugs - live in the layer around it: variant handling, network reduction, normalization, and metric design. Our backwards-winner bug wasn't in TRIBE, it was in how we compared its outputs.
• Contract-first parallel development is a superpower for a four-person, 36-hour sprint. Freezing one CONTRACTS.md (the Score Object + the API) in hour one let everyone build against mocks and snap it together at the end — zero renaming, minimal blocking.

🚀 What's next

Auto-generated video edits (music swap → reorder → pacing), tested the same way; deeper personalization as Backboard learns what wins for each creator; and fine-tuning on hand-validated examples to tighten scores against real short-form outcomes (plus porting our shared-scale scoring into the live product path).

Built With

• auth0
• backboard.io
• base44
• elevenlabs
• gemini
• mongodb
• python
• react