Motion Lore

AI-powered subtitles that translate ballet's silent gestures into stories everyone can understand.

Comment

Inspiration

A YouTube Short of a ballet performance went viral. The comments said everything:

"I would enjoy ballet a LOT more if it had subtitles like this."

"Tbh I wouldn't have understood what was going on without the subtitles lmao"

"Same. I'm too autistic and don't understand without these subtitles"

"Never have I once been any ounce interested in ballet. this has completely changed it for me"

The video had hand-added captions translating the dancers' movements into plain language. People who had never cared about ballet in their lives were suddenly hooked. People who'd struggled to follow the wordless storytelling finally had a way in.

We wanted to build the tool that makes that automatic — no hand-captioning, no guesswork, just drop in any ballet video and get a subtitle track that reads the choreography like a story.

(Yes, some people in the comments argued that subtitles constrain interpretation — that the whole point of ballet being averbal is to let the audience bring their own reading. Fair point. We just think access comes first.)

What it does

Motion Lore translates ballet movement into synchronized narrative subtitles. Drop in a YouTube URL or upload a video — we return a subtitle track that explains what's happening on stage in plain language, so anyone across cultures, languages, or neurotypes can actually follow along.

How we built it

Two-pass LLM pipeline under the hood:

Groq (llama-3.3-70b) first reads the video title, identifies the ballet, and enriches the context — characters, plot, setting. That context then gets injected into Gemini 2.5 Flash's prompt before it ever sees the video, so subtitle generation is grounded in the right narrative from frame one instead of hallucinating blind.

For infra:

  • DynamoDB — two tables. ballet-subtitles is content-addressed by SHA-256 hash of the video URL. Same video uploaded by a million users = processed once, served forever. ballet-jobs tracks async job state (queued → processing → done → failed), persisted so nothing dies on restart.
  • S3 — temporary storage for user-uploaded videos, deleted once Gemini processes them. YouTube URLs skip S3 entirely and go straight to Gemini's native URL handler.
  • FastAPI on AWS — stateless, horizontally scalable, zero server-side sessions.

Challenges we ran into

  • Gemini can't pull from S3 directly — solved by routing uploads through the Gemini File API
  • In-memory job state doesn't survive restarts or scale across instances — moved everything to DynamoDB
  • Bare file hashes were poisoning Gemini's prompt with zero context — added a Groq title classification gate to enrich before analysis

Accomplishments we're proud of

  • A two-pass pipeline that actually improves subtitle quality over single-model approaches
  • Generate-once caching — zero redundant Gemini calls at scale, ever
  • Fully async, AWS-native backend with persistent job state

What we learned

  • Gemini's native YouTube URL handling is genuinely underrated — no download pipeline needed at all
  • Content-addressed caching (SHA-256) is a deceptively simple but powerful dedup strategy for AI workloads

What's next for Motion Lore

  • Support for opera and classical Indian dance forms
  • Community subtitle ratings to fine-tune future generations
  • Embeddable widget for ballet company websites
Share this project:

Updates