LipRead AI — Devpost project page

Inspiration

Most speech recognition assumes there is audio. That assumption fails in more places than people realise: archive footage with damaged or missing tracks, CCTV with no microphone, footage recorded from behind glass, and videos where the audio is drowned out by wind or machinery.

In all of those cases the mouth is still visible and still moving. The information is there, just in a channel nobody's default pipeline reads.

LipRead AI is about recovering speech from the visual channel alone.

What it does

Upload a video with no audio track (or ignore the audio), and get a text transcript from the speaker's lip movement.

• Works on silent video — no audio needed at all
• Produces a transcript with timing
• Built for accessibility, archive recovery, and footage where audio is unusable

How I built it

Visual speech recognition is a sequence problem: you need per-frame mouth features, and you need them aligned to the speech units they correspond to.

1. Face and mouth localisation. Detect the face per frame, then align it to a canonical pose so the mouth region has a consistent size and orientation. Without alignment, head movement alone destroys accuracy — the model ends up learning "head moved" instead of "mouth shape changed."

2. Mouth region extraction. Crop the mouth area from the aligned frame to a fixed resolution, converted to grayscale. Colour adds nothing here and doubles the input size.

3. Sequence model. The cropped frames feed a temporal model that maps the mouth-shape sequence to character or subword units. The output is deliberately over-generated — VSR models are far better at producing candidate sequences than at deciding which one is right.

4. Language-model correction. A second pass scores the candidate sequences with a language model and picks the fluent one. This is where most of the usable accuracy comes from. Visual-only decoding is roughly guesswork on ambiguous phonemes that look identical on the lips (p/b/m are the classic case); the language model is what resolves them.

Everything runs as an async job queue because inference is slow relative to upload, and users should not hold a connection open for minutes.

Challenges I ran into

Homophenes are a hard ceiling. Sounds that look identical on the mouth cannot be distinguished visually — no amount of model capacity fixes it. Realising this early stopped me from chasing accuracy gains that were physically unavailable. The language model is not a nice-to-have, it is the mechanism that makes the output readable.

Head pose kills accuracy faster than anything else. My first version worked on near-frontal footage and fell apart the moment the subject turned. Adding the alignment step before cropping mattered more than changing the model architecture.

Frame rate assumptions are wrong. People assume video is 30fps. Broadcast archive is 25, phone footage is often variable frame rate, and animation is whatever the creator chose. I had to normalise the timeline before feature extraction, otherwise the temporal model sees inconsistent speech rates.

Accomplishments that I'm proud of

• Making silent footage produce readable transcripts at all. The novelty of the input is what makes this interesting.
• The two-stage design: over-generate, then language-model correct. It is a simple idea that carries most of the accuracy.
• Getting the job queue to handle long videos without timing out or losing partial work.

What I learned

Some accuracy limits are physical, not computational. I spent time trying to improve a number that was bounded by the information available rather than by my model. Learning to identify which constraint you are hitting — data, model, or physics — saved me from months of wasted effort.

The visual channel is genuinely different information, not a degraded version of audio. I started thinking of lip reading as "bad audio." It is not. It is a separate signal with its own failure modes, and treating it that way made the pipeline design much clearer.

What's next for LipRead AI

• Audio-assisted mode. When audio exists but is noisy, fuse both channels instead of ignoring audio completely.
• Speaker diarisation. Multiple faces in frame currently produce one confusing transcript; splitting by speaker is the obvious next step.
• Confidence output. Mark which words are visually unambiguous and which were inferred by the language model, so users know what to trust.
• Subtitle export. Go straight from transcript to SRT with timing, since that is the actual deliverable for most accessibility use cases.

Built With

• react