How we built it We built the system using a combination of powerful open-source technologies. The backend is powered by Python, chosen for its extensive libraries for media processing and machine learning. We use:
OpenCV to process the video, extract individual frames, and perform image pre-processing to isolate the subtitle regions and improve text clarity.
Tesseract OCR (and potentially EasyOCR for broader language support) as the core engine to recognize and digitize the text from the processed frame images.
FFmpeg for handling various video formats and extracting audio/timing information efficiently.
A custom algorithm to detect changes between frames, identify subtitle events, and accurately timestamp the beginning and end of each line. The user interface is a simple web application built with Flask or Django, allowing users to upload their files and download the results.
Challenges we ran into The primary challenge was the variability in hardsubs. We had to overcome several obstacles:
Diverse Fonts and Styles: Subtitles come in countless fonts, colors, sizes, and may have outlines or drop shadows, which can confuse standard OCR engines. We had to develop robust image pre-processing steps (like binarization, noise reduction, and color filtering) to create a clean, high-contrast image of the text for the OCR to read.
Dynamic Backgrounds: Subtitles appear over complex and moving video scenes. Differentiating the subtitle text from the background required sophisticated detection algorithms.
Timing Accuracy: Pinpointing the exact frame a subtitle appears and disappears was crucial for proper synchronization. We had to fine-tune our change-detection logic to avoid being triggered by normal video motion while being sensitive enough to catch quick subtitle changes.
Processing Efficiency: Analyzing every single frame of a long video is computationally expensive. We implemented logic to intelligently skip frames where no subtitles are present, significantly speeding up the process.
Accomplishments that we're proud of We are incredibly proud of achieving a high level of accuracy (over 95% on clear source material) in the extracted text and timing. We successfully created a processing pipeline that is significantly faster and cheaper than manual transcription. Another key accomplishment is the system's resilience to different subtitle styles; it can handle a wide variety of colors, fonts, and positions without requiring manual configuration from the user.
What we learned This project was a deep dive into the practical applications of computer vision and OCR. We learned an immense amount about the nuances of video encoding, the challenges of text recognition in "in-the-wild" scenarios, and the importance of image pre-processing for achieving accurate results. We also learned how to optimize algorithms to handle large files and long processing times, balancing speed with accuracy.
What's next for Build the hardsub system that cover minimum fee The journey isn't over. Our next steps are focused on expansion and improvement:
Multi-Language Support: Improving the OCR engine's ability to recognize a wider range of languages, including those with non-Latin characters.
Automatic Translation: Integrating a translation API (like Google Translate or DeepL) to offer an option to automatically translate the extracted subtitles into a language of the user's choice.
Cloud-Based Service: Developing a scalable, cloud-based version of the tool to handle more users and larger files without requiring users to run the software on their own machines.
UI/UX Improvements: Enhancing the user interface with features like a live preview and an in-browser editor for correcting any OCR errors before downloading the final file.
Log in or sign up for Devpost to join the conversation.