Sona

Sona is a voice coach you practise English by talking to. She scores your pronunciation from the audio itself, then a background agent reviews whether her last plan worked and writes the next one.

Comment

Inspiration

I am not a native English speaker, and I am studying IT, where I struggle a bit in English classes, specifically with speaking. Listening and reading are no problem. Writing is a little weak, although these days AI writes the nice sentences for you, which is its own problem if you actually want to improve. But none of that compares to speaking, because I never had the chance to practise it. There is nobody I can speak English with every day.

The way I see it, learning to speak properly means being in a situation where German is not an option and English is the only way through. That situation never came up for me, so I started looking for an app or an AI that could create it.

There are already good apps out there. But I had wanted to build a voice assistant myself for years, and before AI that was close to impossible. Now it worked. I did not want to build another generic assistant, since there are plenty of those, so I built the one that solves my own problem.

What it does

You talk to Sona in English about whatever you want, and she keeps the conversation going.

The first session is mostly her working out who you are. She needs around two minutes of real speech before she will grade anything, and what she measures gets compared against a dataset of 5,000 utterances that human experts rated first. You can tell her which topics you are interested in, but she also picks up interests on her own, like what you are working on or what is going on in your life, and she remembers them for next time.

The part I would show a friend first is what happens between sessions. Every 15 minutes a background agent checks whether anything changed for any learner, and if it has, it writes that person's next session plan by itself. When you come back, Sona opens with what she decided to work on. That is the piece I care most about, because every session teaches her a bit more about what you struggle with, and she adjusts on her own.

After a session you get a summary with real metrics like pace and fluency, but only if you spoke long enough for the numbers to mean anything. Under about 30 seconds she shows nothing rather than guessing.

I also keep two kinds of number apart on purpose. Pronunciation and fluency are measured from the audio. Grammar and the notes she keeps about you are a model's judgement, and the interface says which is which. I did not want them blended into one confident looking score, because that is exactly the kind of number I never trusted in other apps.

She only ever sees your own data. There is one shared collection of conversation topics that grows as people talk, but it holds no personal content, just generic subjects, so everyone gets more interesting things to talk about.

How I built it

The hard parts of the real-time audio come from LiveKit's agent starter template. I had tried that framework before and failed. This time it worked, after some fighting.

The model is Gemini 3.5 Flash. The decision that took longest was how to
handle voice at all. There is a Gemini model that understands speech
directly, but not at 3.5, and the rules required 3.5 or newer, so I could
not use it. What I ended up with is a standard pipeline. Speech to text and text to
speech run on LiveKit, with Gemini doing the reasoning in the middle. I
expect to swap both of those out later.

Getting the grading to work was the other long fight, because it has to come from actual measurements rather than a model's opinion.

What I am not happy with, and shipped anyway. The exercises are the weakest part. They exist and they get stored, but Sona is not good enough yet at bringing them into a conversation naturally. Sometimes she does something that looks like an exercise in plain text instead of opening the real one. She should not need to announce an exercise at all. She should find her own way to push your English up. Connected to that, her detection of what you are actually struggling with is not reliable enough yet. The grading is also only partially fact based rather than fully, and I would like more of it to come from real measurement than it currently does. The numbers still have normalisation problems too, so they are not always as useful as they should be. And the dataset is people reading, not people having a conversation, which is not what this app is for. I looked at better datasets, but the licensing did not fit in the time.

Data sources. speechocean762, which is 5,000 English utterances from non-native speakers, each rated by five human experts, used to calibrate the scoring on a speaker-disjoint holdout. A minimal-pair bank of 2,569 pairs across 441 contrasts, built from the words actually spoken in that dataset. Grounded search through the Gemini API, used in the background job to research conversation topics. And the learner's own audio, which is measured in memory and never stored. I looked at the Speak and Improve Corpus 2025 and did not use it, because it needs a licence application rather than a download.

Challenges I ran into

The one that nearly ended the project was the stupidest. LiveKit's API kept rejecting me and I could not see why. I made a second account, thinking the first one was broken. Same failure. It turned out that my system clock was 60 seconds fast, so every token I generated looked invalid. Nothing in the error said so. I lost about an hour on it and genuinely thought at one point that it would never work, and then the fix took seconds.

The other thing I underestimated completely was deploying to Google Cloud. Locally everything ran fine. But this is not one application. It is a frontend, an API, a background job, and a GPU instance doing the scoring, all needing the right access to my Gemini and LiveKit keys. It took far more attempts and far more time than I expected. Cloud deployment is not something to leave until the end, which is exactly what I did.

Accomplishments I'm proud of

That I can genuinely have a conversation with it, and that it has already helped my English. Not a demo conversation, a real one.

That Sona works out my interests and what I am working on by herself and remembers them. Tools like ChatGPT or Claude can do that too, but I built this one from scratch, and that is a different thing.

Most of all the autonomous part. Every 15 minutes it checks whether anything changed for anyone, and writes that person's next plan without being asked. I had never built anything that runs on its own like that, and it works.

And one thing I deliberately did not do. I stopped pouring time into the grading. It is okay rather than good, and making it properly good needs more time than a hackathon week. I would rather say that than pretend.

What I learned

I thought teaching an agent to judge how well someone speaks would be straightforward. It is the hardest thing in the project.

The biggest thing I learned is the architecture itself. Almost every voice assistant is the same pipeline. Speech to text, then a language model, then text to speech. Which means that by the time the model sees your words, how you said them is already gone. The pauses, the hesitation, the pace. Gemini 3.1 has a model that takes voice directly, but I could not use it under the rules, so keeping that information alive through a normal pipeline was the real difficulty. Building a simple voice assistant is genuinely not hard any more. Building one that hears how you talk instead of just what you said is a completely different problem.

I also learned that cloud deployment across several moving parts takes far longer than running the same thing locally. As an IT infrastructure student, getting all of it onto Google Cloud with a real GPU is the part I am most pleased about.

And what I would tell myself on day one is to start earlier, and not leave the bulk of it to the last day.

What's next

I want to run it myself. Not just the framework but the speech models too, probably something from NVIDIA, and possibly Gemma locally for the reasoning. Right now a lot still happens on LiveKit Cloud, and I would like that to stop.

Then the problem I only half solved, which is hearing how something was said rather than only what was said.

After that, tools. A daily email, a to-do list, a GitHub connection, so Sona knows what I am working on. Better exercises and learning material I can use away from the app. UI polish and a real login instead of an anonymous id. Possibly open sourcing it, once it runs locally.

The end goal is a local assistant on something like a Raspberry Pi with a microphone, sitting in the room. An Alexa or Siri that is a personal English tutor instead of a smart-home remote. The reason it has to be local is privacy. When you actually talk to it every day, a lot of personal things come up that you would not say anywhere else, and those should stay in your own home.

I will be using it every day to practise, which is also the fastest way to find what is broken. I am the person it was built for.

Built With

  • artifact-registry
  • cloud-run
  • cloud-run-jobs
  • cloud-scheduler
  • docker
  • fastapi
  • firebase-auth
  • firebase-hosting
  • firestore
  • gemini-3.5-flash
  • gemma-4-31b-it
  • google-genai
  • livekit-agents
  • python
  • pytorch
  • react
  • scikit-learn
  • secret-manager
  • speech-recognition
  • tailwindcss
  • transformers
  • typescript
  • vite
  • wav2vec2
  • webrtc
Share this project:

Updates

Submission history