RealTalk

🌟 Inspiration

Learning a new language can be overwhelming, especially when trying to recall past lessons in real-world situations. We wanted to create a solution that not only helps users communicate with their surroundings but also retrieves and builds upon their past learning experiences. The idea of combining voice interaction with advanced AI technologies like Retrieval-Augmented Generation (RAG) came from the need for a more personalized, immersive, and dynamic approach to language learning.

💬 What it does

RealTalk is a voice assistant that enhances language learning by:

Translating speech and objects in real-time, allowing users to communicate with their surroundings.
Creating flashcards from user interactions to reinforce vocabulary and phrases.
Retrieving past language learning experiences using RAG, offering tailored assistance based on what the user has previously learned.
Tracking user progress to provide personalized language tips and suggestions for improvement.

🛠️ How we built it

We built RealTalk using:

RAG (Retrieval-Augmented Generation) to recall past interactions and tailor responses based on the user's language learning history.
Speech-to-Text and Text-to-Speech APIs to enable voice communication and translation.
Cohere to enable the assistant to respond to, teach and search through our database for the most relevant memories.
Convex we leveraged Convex's suite of backend API services to query and send data to our database, supercharging our work.

🚧 Challenges we ran into

One of the biggest challenges was integrating RAG with real-time voice input to retrieve relevant past experiences without causing delays. We also faced issues with ensuring that translations were contextually accurate, as direct translations often lacked nuance. Another challenge was tracking user progress and storing interactions in a way that allowed for seamless recall and updates.

🎉 Accomplishments that we're proud of

We are proud of successfully integrating RAG into a voice assistant with the help of Cohere Rerank in a way that enhances personalized learning without the complexities of a vector database. The creation of dynamic flashcards from real-world interactions was a key milestone, as it provides users with a more engaging and effective way to retain new vocabulary.

📚 What we learned

We learned a lot about the challenges of real-time voice interaction and language translation in an AI context. Understanding how to use RAG effectively to recall past learning experiences was a key takeaway, as it added depth to the user’s learning experience. We also gained insights into how to personalize learning tools to make them more effective for users at different stages of their language journey.

🚀 What's next for RealTalk

Moving forward, we plan to:

Expand support and integrate advanced language models for better translations and nuanced responses.
Enhance*contextual learning experiences* by allowing the assistant to recognize physical objects and environments in augmented reality with wearable glasses.
Introduce gamified elements to make the learning experience more engaging and fun.
Improve speech recognition and response times, making interactions even more fluid and natural.