Catchup Captions

Inspiration

Real-time captions make spoken information visible, but they do not always help users recover after they miss a sentence or lose the context of a conversation. A single missed sentence can make the explanations, examples, or decisions that follow much harder to understand, causing users to fall further behind even while captions continue to appear.

What it does

Catchup Captions converts spoken conversation into live, readable captions, helping deaf and hard-of-hearing users follow lectures, meetings, discussions, and everyday conversations in real time. Its core feature goes beyond standard live captioning. If a user loses track of the conversation, they can tap the “I missed that” button. The app captures the recent transcript and provides a concise summary of the key context, allowing the user to quickly understand what they missed without manually searching through previous captions. While the summary is being generated, live captions continue to appear. This prevents the user from missing even more of the ongoing conversation while trying to recover the earlier context. Catchup Captions helps users not only see what is being said, but also regain the flow of the conversation and continue participating.

How we built it

We built Catchup Captions with React and TypeScript and used browser-based speech recognition to convert spoken audio into live captions. As new captions appear, a rolling transcript buffer continuously retains approximately the last 30 seconds of conversation. When the user taps “I missed that,” the app creates a fixed snapshot of the transcript stored at that moment. This prevents newly arriving captions from changing the content being summarized. The snapshot is then sent to the OpenAI API, which extracts the key context and returns a concise, easy-to-understand summary. The summarization process runs separately from live captioning, so new captions continue to appear while the summary is being generated. We also added safeguards for speech-recognition failures and unclear transcripts, allowing the app to provide clear feedback instead of failing silently or generating unsupported information.

Challenges we ran into

At first, we tried summarizing the recent conversation while new captions were still being added. Because the transcript kept changing during the summarization process, the scope of the summary could become unclear and no longer match the exact moment the user had missed. To solve this, we fixed the summarization window to approximately the 30 seconds before the user tapped “I missed that.” The app creates a snapshot of that specific section and summarizes only the captured text, ensuring that the result reflects the context the user actually wanted to recover.

Accomplishments that we're proud of

We are proud that Catchup Captions is a working prototype rather than only a visual concept. With real microphone input, the app can generate live captions, retain the most recent conversation, create a one-tap AI summary, and continue displaying new captions while that summary is being generated. We tested the complete flow using both podcast audio played aloud and live speech read directly into the microphone. In both cases, the app successfully captured the spoken content and generated a summary of the recent context. We also verified that the project passes TypeScript checks and production builds.

What we learned

I learned that improving accessibility does not always require adding more features or creating a complicated new workflow. Sometimes, the most meaningful improvement comes from identifying what an existing experience still fails to support. Live captions already help users see spoken information, but they offer limited support once a user loses the context. Instead of replacing the familiar captioning experience, I focused on filling that gap with one simple action: letting users recover the recent context by tapping a single button. This taught me that an effective accessibility tool should reduce effort and solve a specific problem at the exact moment help is needed.

What's next for Catchup Captions

Our next priority is expanding Catchup Captions beyond English. We plan to support languages such as Korean and Spanish so that more deaf and hard-of-hearing users can follow conversations and recover missed context in their preferred language. We also plan to test the app with deaf and hard-of-hearing users in real lecture, meeting, and everyday conversation settings. Their feedback will guide improvements to summary clarity, caption readability, and interaction design. In the future, we hope to develop an installable mobile version that offers easier access from a user's home screen.

Built With

• openai
• react
• typescript
• vercel
• webspeechapi