Logo
View of UI
Live transcriptions

CoverMe Project Story

Inspiration

CoverMe was born out of the recognition that many people struggle in social interactions, particularly when it comes to maintaining a fluid and confident conversation. Social anxiety, awkward silences, and the fear of saying the wrong thing often make navigating these situations difficult. We wanted to create a solution that could assist individuals in real-time, helping them feel more confident and capable in their social encounters. CoverMe serves as an in-the-moment coach, offering suggestions and guidance, eventually helping users develop confidence on their own.

What it does

CoverMe is a real-time social assistant powered by AI. It listens to conversations through an earbud and provides intelligent, context-aware suggestions on what to say next, when to ask a question, or how to keep the conversation flowing smoothly. The application is designed to help individuals with social anxiety or those struggling to maintain conversations by offering discreet, timely guidance. Over time, the goal is for users to become more self-reliant, as they internalize the AI’s suggestions and gain confidence in their own social abilities.

How we built it

CoverMe was built using a combination of various technologies and frameworks. For the backend, we used Whisper by OpenAI to convert live audio into text, forming the foundation for transcription and analysis. To differentiate between speakers in the conversation, we utilized PyAnnote and Resemblyzer, with Resemblyzer being key in identifying who was speaking and filtering out the user's voice. We used Gemini to generate conversational suggestions based on the context of the dialogue, and for converting those suggestions back into speech, we integrated Eleven Labs. We built the frontend using JavaScript, ensuring a smooth and dynamic user experience. The frontend allows users to start and stop listening via simple buttons, and the conversation history dynamically updates in real-time with transcriptions and suggestions. Flask was used to handle communication between the frontend and backend, enabling real-time updates through Server-Sent Events (SSE)

Challenges we ran into

Real-time transcription: Ensuring the transcription system was fast enough to provide real-time suggestions was a challenge. We initially had issues with audio chunking, but switching to Whisper's incremental streaming helped mitigate this.
Speaker diarization: Identifying who is speaking at any given time during a conversation was difficult. We had to replace PyAnnote diarization with Resemblyzer to filter and identify speakers more accurately.

Accomplishments that we're proud of

Real-time conversation assistance: We successfully built an app that provides real-time, relevant conversational suggestions based on actual conversation data.
Speech-to-text accuracy: Achieving high transcription accuracy with Whisper AI in real-time was a major milestone, allowing seamless interaction.
User-centric design: The app’s interface is clean, minimal, and easy to use, making it accessible for users with social anxiety or those who are just looking to enhance their conversation skills.

What we learned

Real-time AI integration: We learned the challenges and complexities of integrating real-time AI into an app, especially in terms of maintaining system performance while ensuring accuracy and usability.
Machine learning optimization: Fine-tuning models like Whisper and Resemblyzer for our specific use case provided valuable insights into optimizing machine learning models for mobile applications.

What's next for CoverMe

Expand AI capabilities We plan to enhance the AI's ability to provide deeper, more personalized suggestions based on user preferences and conversation history.
Multi-language support: We're working on adding multi-language support, allowing users from different regions to benefit from the app.
Social feature integration: In future iterations, we want to add features like sharing conversation tips with friends or inviting people to practice social scenarios within the app.