Soundscape

Inspiration

Captions show every word but miss the door slams, the phone ring, the glass breaking, and the keys dropping. People who are hard of hearing have a gap in knowledge of what’s happening around them, with current solutions inaccessible to many. Soundscape captions the sounds that matter, using all the context available.

Soundscape is a video game-inspired tool that transcribes non-speech audio into captions, using visual and environmental context to help identify 500+ sounds, giving people who are deaf or hard of hearing more ways to experience sound.

What it does

1. Upload a file to the dashboard or use the extension on a YouTube video.
2. It samples images and audio to determine where each part of the video takes place.
3. It detects sounds such as traffic, animals, and alarms, then uses current environment to help choose the most fitting labels.
4. It turns those sounds into captions that appear when the sounds happen.
5. The extension directly overlays the video with finished captions alongside combined with directional information

How we built it

1. Environment analysis

• FFmpeg and FFprobe extracts audio, video frames, and timing information
• PySceneDetect and OpenCV divide the video into scene spans labelling each one to an environment
• SigLIP and CLAP, loaded through Hugging Face Transformers and PyTorch, compares frames and audios to each environment
• NumPy fusion and YAML settings combine visual/audio scores on a custom weight split to select an environment, while falling back to general if confidence is low

2. Sound detection

• SoundFile and torchaudio prepare overlapping audio windows for the models
• PANNs CNN14 scores AudioSet sounds, which the project maps to its caption vocabulary.
• Silero VAD identifies speech regions
• Python post-processing applies custom environment priors (weightings), filters uncertain events, and produces baseline and conditioned results

3. Captions and viewing

• Python caption writers turn detected events into timed WebVTT tracks and can merge them with supplied speech subtitles
• FastAPI and Uvicorn provide the local upload, processing-status, video, and caption endpoints
• HTML, CSS, and JavaScript power the user side while a Chrome Manifest V3 extension overlays conditioned captions on live YouTube playback
• yt-dlp, Node.js, and FFmpeg get supported YouTube videos for the local backend along with direction analysis added to the caption display

4. Evaluation and tuning

• Versioned JSON contracts and YAML configuration define how the pipeline stages exchange results and how models are configured
• AudioSet strong-label clips and ESC-50 provide evaluation and tuning data, with Kaggle tooling used to obtain the AudioSet dataset
• The Python harness, Optuna, and MCP tooling support repeatable scoring and parameter searches
• pytest, Node's test runner, and Playwright check the Python pipeline, extension logic, and browser behavior

Challenges we ran into

-Making captions accurate in early stages of testing, the model (PANN) used to classify sounds produced many plausible but wrong labels of the events happening. We improved accuracy by introducing environment based scoring, class thresholds, and safety exceptions.

Accomplishments that we're proud of

• Real-time processing We built the project with the use of captioning sound events in real-time as the goal. Our current algorithm uses all context available and would be more accurate with live feed because of additional location like location data.
• Custom environment priors We have a dozen preset environments used to better classify sound events based on the current scene in the video. Each environment has tuned sound coefficients for accuracy based on a labelled dataset ran through a pipeline to determine which sounds are more likely to be heard in a certain environment, so it is based on real data and not generated!
• In-house processing: All of the sounds events being classified and the video segmentation with environment labelling is being processed locally on lightweight ML models that were fine-tuned through video datasets and manual inspection.

What we learned

From building Soundscape, we had to learn about common pitfalls when dealing with sound and captioning:

• Scene Context: Sounds are easier to classify and interpret when knowing what is happening around in the video and related context
• Sounds categorized: Having more sounds with specific information could be counterintuitive to what we're trying to achieve, and having a threshold and a broader sound is more helpful than an inaccurate caption
• Evaluation data: Datasets of videos we looked at for training the environment coefficients was hard to find and often needed manual correction
• Synchronizing: The entire process of video formatting, processing video and audio tracks, caching, and displaying it back all had to work in hand to provide the sound captions at the correct timestamps

What's next for sound thing

• Connecting with hardware: Since we built the processing with real-time in mind, we would love to get our hands on a pair of smart glasses with a camera to test the performance on a live-streamed feed.
• Custom Mobile Software: To stay aligned with the goal of allowing this project to remain accessible, we plan to implement a mobile app which can connect to a local (or through the internet) server so that many others can use this project without specialized hardware

Built With

• clap
• claude
• codex
• css
• gemini
• html
• javascript
• pann
• python
• siglip