-
This is the architecture of this project
Inspiration
This project was inspired by the challenge of reconstructing speech from silent videos. In many real-world situations, audio may be missing, corrupted, or unavailable, while the visual movement of the lips still contains useful speech-related information. I wanted to explore whether deep learning could learn the relationship between lip motion, facial landmarks, and acoustic speech representations.
What it does
Reconstruct_Speech_from_Silent_Video is an AI system that attempts to reconstruct speech from silent mouth videos. The system takes a video of the mouth region as input, extracts visual motion and lip landmark features, predicts content/speech units, and then generates a mel-spectrogram that can be converted into audio using a vocoder.
How we built it
We built the project using PyTorch and a lip-to-speech deep learning pipeline. First, video and audio data are preprocessed into cached training samples, including mouth ROI frames, lip landmarks, mel-spectrograms, energy features, and speech units.
The model combines multiple components:
- A visual tower to learn mouth appearance and motion from video frames.
- A geometry tower to learn lip shape and landmark movement.
- A fusion module to combine visual and landmark features.
- A Conformer-based encoder to model temporal context.
- A content-unit prediction branch to predict discrete speech units.
- A decoder that uses the predicted or teacher content units to generate mel-spectrograms.
The system is trained using multiple losses, including unit classification loss, mel reconstruction loss, and energy loss.
Challenges we ran into
One of the biggest challenges was that direct video-to-mel prediction often produced average or unclear speech representations. Lip motion alone is also ambiguous because different sounds can have similar mouth shapes. Another challenge was aligning video frames with mel-spectrogram frames, since video and audio features have different time resolutions.
We also had to handle preprocessing, landmark extraction, motion modeling, memory limits during training, and making the model learn useful speech-related content instead of only memorizing average acoustic patterns.
Accomplishments that we're proud of
We are proud of building a complete lip-to-speech pipeline from preprocessing to model training. The project includes mouth ROI extraction, landmark-based motion features, content-unit prediction, mel-spectrogram reconstruction, and a modular model architecture.
A key accomplishment is moving from a simple direct mel prediction baseline to a content-unit based architecture, which separates speech content prediction from acoustic reconstruction.
What we learned
Through this project, we learned more about computer vision, speech representation learning, sequence modeling, and multimodal deep learning. We also learned why lip-to-speech reconstruction is difficult: visual information is incomplete, many phonemes look similar on the lips, and temporal alignment between video and audio is very important.
We also gained experience with PyTorch model design, Conformer-style blocks, speech units, mel-spectrograms, landmark motion modeling, and training/debugging deep learning systems.
What's next for Reconstruct_Speech_from_Silent_Video Public
Next, we plan to improve the quality and stability of reconstructed speech by training on larger datasets, improving the content-unit prediction branch, testing stronger vocoders such as HiFi-GAN, and improving synchronization between lip motion and generated audio.
Future work may also include better evaluation metrics, real-time inference, support for more speakers, and more robust preprocessing for videos with different lighting, face poses, and speaking styles.
Log in or sign up for Devpost to join the conversation.