LingoPilot - AI Voice Language Learning Agent

A low latency A.I conversation partner to supercharge language learning with comprehensible input by improving fundamentals like speaking, writing and reading all in a gamified way!

Comment

Inspiration

When I was learning German/Deutsch all by myself (from scratch to B1) a year ago, I was faced with a lot of problems. There was not a clear path that was designed towards me, all scattered/generic in their own way. Didn't really like the traditional methods that were out there which taught the language from a grammar point of way. Clearly not the way babies learn.

That's when I learnt about learning language through comprehensible input. "Comprehensible input is the language that learners can understand without recognizing all the grammar rules in it." ~ sanako.com. Essentially, showing what the words mean - through images, actions, behavior, etc. out loud. I even started watching a lot of movies/series! - just so I could recognize patterns and gain knowledge from it.

But then I realized, I was at a point where I could understand what the sentence meant - but wasn't able to speak/write by myself. I had a very inconsistent schedule of learning - so couldn't really consider scheduling time to talk with someone via some online websites. I just wish there was a service that would enable me to learn in realtime whenever I wanted / had time.

And this is how LingoPilot was born!

What it does

An A.I Voice first low latency - language learning app that gamifies your journey by honing your speaking, listening, reading and writing skills through comprehensible input at its core! Launching with 2 modes:

  • Character Convos

    Realtime AI Voice agents ready to have conversations with you in the language and difficulty that you choose! Multiple characters, personalities, backgrounds - all available at any time / on-demand that requires no scheduling! And don't worry, they will guide you throughout your way as you learn and make mistakes.

  • Picture Quest

    A mini-game that involves solving questions based off images that you are required to answer in your own way. You can be descriptive, give your own opinions, make mistakes - everything is allowed! LingoPilot understands you are learning and will guide you with reasoning if there are mistakes in the sentence structure, typos and overall grammar. You let your ideas flow first, and learn to fix them later as you go forward!

Its not the final version ~ still acts as a copilot to your "main" learning journey and hones your skills - not for a total beginner at the moment. But soon enough, with more beginner friendly exciting modes - LingoPilot will no doubt become a one-stop destination for all your language learning needs.

How I built it

To build a realtime voice agent, I had a brief yet simple idea of connecting a TTS engine with a LLM. But with traditional LLM providers - the "normal" inference speed being so slow - this method would surely make the user feel bored. On an average it would have taken 5-7 seconds for each response. The TTS engine that I had tested on (Amazon Polly) was almost realtime as I could just stream the audio directly to the client as it starts generating. So how do we make the LLM part faster?

image

Enter SambaNova Cloud. One of the fastest LLM inference providers! Immediately plugged their models - easily via the OpenAI python library. As you can see, from the stats above (best case - worst case) times of both the approaches - the one with SambaNova improved the response times vastly making it being close to realtime responses.

Both the modes utilize the RAG model in various ways. The Character Convo mode - when in action - gets enriched with character data from the database with the previous messages whereas the Picture Quest mode retrieves images using the Pexels API, analyzes and creates a question using (image recognition) Vision models then presents a unified set of images with their questions. The expected_answer is not pre-filled when creating the quest as I wanted to allow the user - freedom to write whatever they want about the image and the question to act only as a pathway - not an objective.

image

image

Here the backend REST API is built with Python + FastAPI which was chosen for its simplicity, the devex/speed and the great support it has for AI related stuff with the immediate support of SambaNova models through the openai library. As always for the frontend I went with Typescript + React + Next.js at the core. Though for the UI part, I tried NextUI instead of shadcn/ui just for fun - also cause' it has more polished/aesthetic components. For handling the auth, I wanted to try out StackAuth (implementing auth is time-taking) as it is opensource and I can self-host it whenever I feel the need to. Now for the database - I chose to go with Postgres and not with No-SQL (although it is unstructured data that I'm dealing with, but I also didn't have a specific reason to choose NoSQL thinking about the data duplication, no complex queries, etc.) To get started, instead of vanilla Postgres where I'd have had to setup an ORM or a query builder - I thought the hosted version of Supabase was convenient (because of their lib) and I can even self-host utilize their Auth/Storage/Serverless Functions down the line. All of these hosted alternatives and component libraries helped me ship this app faster than I could have by implementing and self-managing everything they provide (believe me, I was the guy who thought writing it from scratch was faster / better - as i had better control).

Finally, deployed the frontend on Vercel (fast to deploy) and the backend on an EC2 instance but with CI/CD setup. It is setup in such a way that anytime I push a commit to the default branch, a build will be automatically triggered on the EC2 instance that will rebuild the image using the Dockerfile in the FastAPI app and deploy a container on the configured domain. All of this was easily manageable by using an open-source tool Coolify!

Challenges I ran into

Time. I joined the hackathon and building too late, with an idea which might have been too ambitious given the time constraints I had to work with. Choosing the right features is always a challenge when you have a lot planned initially.

When creating the quest, I tried using the Tool Calling functionality where I'd have told the 70B model to do all the things (getting images, getting a question from the Vision 11B model) using the tools I provided it. But while implementing this, I thought to myself, the 70B anyways has just one task to do and that too in a predefined sequence - i probably don't need the Tool Calling functionality for my use case. Pure code would be faster. The Tool Calling functionality would be useful where the LLM is supposed to do multiple things and take decisions on its own - where we essentially can't define a path for it.

It was when I was tinkering with the Picture Quest verification logic, where I pass in an image (with its question & answer given by the user) and its supposed to give me 2 data points. One being the expected_answer and the second being reason. I tried proompting my way out of this and force it to output only json but it just won't. To fix this, I put a light LLM (3.1 8B) to parse the output for all questions given by the Vision model, cleanse it and return pure json. This worked like a charm!

Initially, to make the UI flow faster for picture quest - on every question's next button - the next question would be generated. But this approach in practice took a lot of time (4-7 seconds) which I thought made the app feel slow. So instead, every time you create a new Picture Quest session - for now the backend makes sure all the questions are ready before it drops the first question to you. This approach does take 15-20 seconds upfront (has a UI loader) but provides a faster UX experience when actually solving the questions.

I was working with some new stuff this time namely - Stack Auth, NextUI and one of these or even both mixed together would bring an issue where the UI would refresh/flicker when there was a useQuery hook involved. I even tried ditching useQuery and firing API calls via useEffect, but still the same issue. Although fixed this issue on some parts of the app by getting the user's access_token inside a function and not at the top-level of a component - it still persists in some areas of the app. Gotta see why this is happening exactly.

What I learned

The hackathon submission date extension was a life saver! I had just finished the Character Convo mode (being the primary juice of LingoPilot) when it was extended. Had a choice of making the existing mode reach it's final version OR introduce the Picture Quest mode (among other modes) which explores SambaNova models even more and solves a lot of problems on the user's side. Just after finishing up Picture Quest and realizing what all the Vision models are capable of, a ton of truly amazing ideas popped up which I unfortunately queued for later implementation (time constraints = hard decisions).

Initially I did face some problems with the SambaNova models but were quickly resolved through their awesome community! I learned that its not me who always has to solve problems on my own but take help from the online community. Of course there are times when I want to take my time, fail and learn from it but there are times where I just want to squash that annoying issue.

A technically unrelated issue would be the unfortunate bad quality of the DSLR camera output after recording the entire trailer/demo video. So the lesson? it would be my duty to check the output of the camera I'm using before using it.

What's next for LingoPilot

After slacking on this idea for months, I finally made it into a reality! Thank you - SambaNova for this amazing opportunity. There is a ton of content that I've planned for the app to make it an ultimate platform to learn languages naturally. Although some stuff that is on priority would be:

  • Beginner-friendly modes: Add more exciting and simplified features to help absolute beginners get started easily, with step-by-step guidance and interactive exercises to build a strong foundation with an even more gamified experience by introducing scoring/leaderboards!
  • Enhanced language grasp: Create new modes focused on understanding cultural nuances, common phrases, and real-world scenarios to help users master the language naturally.
  • Faster voice agent: Work on making the voice agent even quicker by introducing prompt caching and experimenting with WebRTC to achieve near-instant responses.

Optimizations

The Picture Quest questions creation take a considerable amount of time upfront. The next step here would be to generate only one question when the user creates the quest but then as they are solving that one - asynchronously create the next question and so on. This will create amount of questions that are required at that time (maybe the user will quit mid-quest) and thus save extra costs due to unnecessary LLM requests and at the same time deliver a super fast user experience.

But for a new user, who may be trying one of the quick start questions.. I'm thinking of restructuring the behavior in such a way that this user will receive an already generated set of questions from some other user. So basically re-using DB data for an even faster exp. and to prevent unnecessary LLM calls.

To make the Conversation have even more low latency, I'm thinking of caching previous recent messages probably on the UI side or redis - as this is what happens before the actual generation starts.

Built With

Share this project:

Updates

Submission history