Mobile home screen
Mobile dictionary screen
Live conversation with tasks and warnings. Tasks and warnings are computed and updated realtime.
Home screen
Dictionary screen. Words are pulled and tracked automatically from all of the user's conversations.
Full home screen. Jump into a previous conversation to continue where you left off or start a new conversation from a scenario.

LinguaConverse

A platform for practicing both speaking and listening skills in any common language through user created scenarios. Learning is gamified through completing tasks during conversations.

Project Overview

Problem

There are no convenient resources for practicing conversational skills in languages you are trying to learn. This is a huge roadblock to the successful learning of any non-first language, as studying vocabulary and grammar alone will not build strong language skills.

Solution

LinguaConverse solves this by providing a platform for having conversations in an endless number of user-customizable scenarios. It leverages ElevenLabs' Conversational AI for STT, TTS, and the conversational flow.

Scenarios
Scenarios can be customized with a name and a description of the scenario. Tasks may also be added which gives the user a goal when having a conversation. Tasks have a completion metric (completed, progress steps, percentage) and a description so natural language can be used to compute is status in context of the conversation. Scenarios can also have custom aesthetics through background, avatar image, video effects, ambient sound, and accent colour.

Conversations
Conversations can then be started from a scenario. They are immersive through the scenario's aesthetic customization. Subtitles will automatically show for the assistants last message, but these can be hidden to practice without them.
During the conversation the tasks' completion statuses will automatically be computed on the backend and update live on the frontend. Grammar/Language warnings are also checked for, and they are output live to the frontend as well.
Conversations can be paused at anytime. The user can always come back to a past conversation and continue from where they left off. During a conversation all words are saved to a user dictionary. It tracks the number of times the user has used a word themselves, or seen the assistant use a word.

Dictionary
During conversations, words are saved to the user dictionary. Users can view their dictionary at any time. This gives insight in to their strongest and weakest words. This could also be used to highlight in a conversation when a word is completely new to a user.

Technical Details

Github Repo

https://github.com/AidanMacMillan/eleven-labs-hackathon

Key Details

Architected in a way to easily run natural language analysis in realtime on ElevenLab conversations.
- All analysis is behind a Redis driven queue to allow any number of workers to pick up analysis jobs. This enables full horizontal scaling.
Has it's own persistence to enable ElevenLabs conversations to be picked up later in the future even if the call has been ended.
- This is supported through ElevenLabs Conversational AI's custom LLM feature, which point's to the application's own completions endpoint. The application can then hook into the conversation flow in any way it needs to.
Gemini Flash 2.0 was chosen for both the conversation LLM as well as for AI analysis agents for checking task statuses, finding grammar/language errors in user messages, and finding words to save in the user's dictionary. It was chosen due to it's speed and quality.

Tech Stack

Main Application: Sveltekit
- Frontend: svelte, tailwind
- API: Hono, Sveltekit Actions
Database: postgres
- drizzle-orm: used to manage the database schema and provide typescript types to other packages
Conversation analysis: Node
- bullmq: Used for queue and worker management
Cache: Redis
- Only used for the queue
Realtime notifications: Soketi
- Used by both the SvelteKit API, and Conversation Analysis server to push realtime data to the frontend
LLM: Gemini Flash 2.0