Tweezr

Word tweezing & dialogue assembly

Comment

Inspiration

An editor looking for one spoken line scrubs a timeline. Transcript search exists, but it stops at the sentence — and the cut still happens somewhere else.

The agent versions of this are worse. You ask for a rough cut, a file comes back, and you find out it used the wrong take after the render. The decision that matters happened inside a box you could not see.

We wanted the opposite: search down to the individual word, and put the proposal on the editor's screen, over the real footage, with every fragment's source visible — before anything is produced.

What it does

Tweezr turns recordings into a word-level searchable dialogue library and lets you assemble a cut out of it.

  • Search a phrase, get ranked candidates across every take, filtered by delivery.
  • Click any word and exactly that word plays. Shift-click a second word to take the span. This is the namesake: taking three words of a line, not the line.
  • Assemble a new sentence from words spoken in different recordings, at different times.
  • Preview without rendering. The player seeks around the source files, so a rough cut costs nothing until you commit.
  • Bring your own footage, audio or video, any language. Drop a file in, or paste a YouTube link and let it fetch. Either way it is transcribed and searchable in one step.
  • A vocabulary panel shows every word the library can say. Click one to search it.
  • WebMCP tools let an external agent work on the same timeline you are looking at. When it calls commit_render, its promise waits on your approval dialog.

One irreversible step, one approval, and provenance on every fragment.

How we built it

ClickHouse is the retrieval engine, not a metrics bucket. One row per word, primary key (project_id, word_norm, start_ms). Phrase search is a single pass using higher-order array functions — candidate lines come off the index by the phrase's first word, their words are collected into a sorted array, and arraySlice finds every consecutive match. No self-joins, so an N-word phrase is not N table scans. The SQL lives in one file and is asserted equal to a local reference implementation on every test run.

faster-whisper produces word-level timecodes on CPU, multilingual by default. Ingest takes a local file or a YouTube URL through yt-dlp, capped by duration so a three-hour podcast cannot occupy the only instance. Both paths land in the same document contract, so there is one validation and one writer, not two.

Gemini 3.7 Flash does a job the index cannot: one call per take, audio in, a delivery label per line out (calm, tense, whisper...) with a confidence. That becomes a column, so "the calmer reading of this line" is a filter and a ranking rather than a guess.

Google ADK runs an in-page assistant for visitors with no agent of their own. Both entry points — the agent and the panel — mutate the same store, so the timeline the agent reports is the timeline the human is looking at.

FastAPI + vanilla ES modules. No bundler, no build step, one origin, one container on Cloud Run. Anonymous sessions with a credit ledger; only render and ingest cost anything.

Challenges we ran into

A hyphen voided a paid ingest. Whisper's segment text keeps state-of-the-art whole while its word timestamps split it at the hyphens, so our contract check called the take corrupt and threw it away — after the user had paid for transcription. The fix was not the tokenizer. The line text never reaches ClickHouse; the table stores one row per word. We were voiding correct rows over a field we do not store. validate() now returns only what makes a document unusable, and everything cosmetic became a note.

Playback was loading 180 media files a second. The rAF loop notifies the store ~60 times a second and the story track rebuilt every block, recreating each thumbnail <video>. The lists now carry signatures of the things that actually affect their markup.

Accomplishments that we're proud of

  • The millisecond claim is measured, not asserted. A hand-picked 360 ms range rendered to 360 ms of audio, read back off the output file. Zero drift.
  • The SQL is checked against a second implementation. Returning the wrong take quietly is the most expensive failure a search can have, so every scenario runs through both the ClickHouse query and a local reference and the results are compared.
  • We did not give the agent word-level control, on purpose. It proposes lines; the human refines to the word; get_timeline_state carries the refinement back. Word-level picking is a judgement that takes ears, and handing it to a model means putting the whole transcript in a tool result to guess at it.
  • Uploads are isolated by a project id the caller cannot supply. It is derived from the session cookie, so the isolation does not rest on a validation rule we could get wrong.
  • 602 tests, plus doctests, plus a deploy checker that runs 31 checks against the live URL — including whether the interface has anything for a first-time visitor to click.

What we learned

The severity of a check matters more than the check. Our validation was correct and still wrong, because it was fatal about a field that never reaches the database.

A search-first interface is a memory test. It works on a corpus you know and fails on footage you just shot, because the transcriber does not always hear what you said. An empty result then reads as a broken product rather than a wrong guess. Showing the vocabulary was the fix, and the complaint that produced it was one sentence long.

Where the human belongs is a design decision, not a limitation. A model is good at "which of these takes reads calmer". It cannot hear that the cut should start half a word later. Drawing that line deliberately made the agent surface smaller and the product better.

What's next for Tweezr

  • Fill a missing word with synthesis. When the library does not contain the word a sentence needs, generate it — Chirp 3 HD voice cloning to match the speaker already on the timeline, so the joined line still sounds like one person. The generated fragment enters the timeline through the same segment shape as a recorded one, but the provenance strip labels it synthesised rather than naming a source file. That distinction is the product's spine and it does not bend for a convenient feature.

  • Fill a missing shot the same way. A word can exist in audio with no usable picture — the speaker turned away, the camera cut, the take was audio only. Veo generates the missing frames from the surrounding shots and Imagen fills a still, so the cut plays without a black gap. Marked as generated, in the strip and in the approval dialog, with the recorded and generated durations reported separately before the render.

  • Let Gemini compose, not just retrieve. Today the agent finds lines and the human arranges them. The next step is asking for an outcome — "a forty second cut that opens calm and ends on the hardest reading of this line" — and having the model plan the composition: which takes, in which order, where the beats fall, which fragment is missing and would need synthesising. It still arrives at the same approval dialog, and the plan arrives with its reasoning attached, so the editor is judging a proposal rather than accepting an output.

  • Signed URLs on object storage for uploaded media. The current mount is unguessable but not access controlled, and we would rather say that than hide it.

  • A global spend brake, not just per-session credits.

  • Speaker diarisation, so "find where she says it" is a filter.

  • Larger Whisper models on GPU, and an alignment pass so long words stop drifting.

Built With

Share this project:

Updates

Submission history