Verba

UI of Verba
Lens Studio's face mapping capabilities

Inspiration

As a team of first-generation Americans and ESL speakers, we’ve experienced firsthand how difficult it is to learn English. Progress in learning often doesn’t match the pace of real-world use, and one of the most frustrating experiences is knowing what you want to say, but not how to say it.

Over 1.5 billion people worldwide are learning English today (Kent State University). For many of them, the gap between understanding and expression is a daily reality. It leads to hesitation, loss of confidence, and in many cases, silence.

One of the biggest barriers in language learning is pronunciation. Reading and writing can be practiced individually, but speaking requires physical control. Learners must know how to move their tongue, shape their mouth, and produce sound correctly.

Today, learners often rely on finding others who are willing to take the time to carefully correct their pronunciation, which can feel uncomfortable and discouraging. Others listen to recordings and try to emulate the phonemes, but receive no real-time feedback. When these options fall short, learners turn to speech coaching, which can be expensive and inaccessible. As a result, many learners plateau. They understand the language but cannot confidently speak it.

We believe language learning should empower people to be understood. In response to this gap between learning and application, we created Verba.

What it does

Verba uses visual and audio detection to provide real-time, personalized feedback on how to correctly pronounce words. Users are given a word to say, record their attempt, and receive immediate guidance based on both their speech and facial movements.

For example, if a user is given the word “coding,” they might pronounce the first vowel as an “oh” sound instead of an “aw.” Verba detects this difference and tells the user precisely what physical adjustments to make, e.g. "Drop your jaw more for the AW sound" and will visually show when a users mouth is in the correct shape. This gives learners a virtual speech coach tailored to their specific needs. We package this experience in a fun, engaging UI where users earn rewards based on how quickly they improve, with real-time visual cues on their own face to guide them along the way.

How we built it

To build Verba, we used Lens Studio and JavaScript. After extensive work in the ideation and algorithm design phase, we realized that the best approach was a two-stage system.

First, Verba presents a word to the user, and through testing and research, we identify the key sounds that make up that word, also known as its phonemes. For example, for the word dog, we can break it down into: $$ dog = d + ah/aw + g $$ From there, we use Lens Studio’s built-in audio detection capabilities, combined with visual detection using face mesh, to determine how close a user is to correctly pronouncing the word. Using the face mesh tool, we track the coordinates of different facial features, such as the corners of the mouth, the top lip, and more.

Through extensive self-testing, we mapped how facial features change when saying the word "dog" and identified quantifiable metrics to evaluate pronunciation accuracy. One such metric is the distance between a user’s top lip and bottom lip, calculated by: $$ Mouth Size = Top Lip - Bottom Lip $$

We then test to determine the minimum Mouth Size required to correctly produce the "ah/aw" sound. To account for differences in distance from the camera, we normalize this by calculating a ratio between face size and mouth size. Applying this same process to other phonemes in the word, we combine these visual metrics with Lens Studio’s audio detection to quantify overall pronunciation accuracy. For example, if a user’s mouth size ratio for the "ah/aw" sound falls within 5% of our benchmark, it is marked as 100% accurate. This allows for slight natural variation in pronunciation and tone. A user’s final accuracy score is then aggregated across all phonemes, highlighting exactly which parts of the word need improvement.

If we detect that the accuracy for the "ah/aw" sound is low, we instruct the user to open their mouth more and provide real-time visual cues to guide them on how much to adjust.

Finally, we drew inspiration from social media and gamified the experience into a fun, engaging system that users can share. The faster a user correctly pronounces a word, the more points they earn, creating opportunities for friendly competition among learners.

Challenges we ran into

Our team ran into two major challenges during development.

1. Calculating Accuracy
Lens Studio provides a limited library of words it can detect, along with a probability score for how likely it is that a user said a given word. However, many words share similar sounds, and variations in accents made it difficult to rely on this probability alone as a measure of pronunciation accuracy.

This created a key problem: we needed a way to evaluate how well a word was pronounced, not just which word was detected. Our initial approach relied too heavily on Lens’ probability outputs, which were not granular enough and often misrepresented user performance.

To address this, we shifted toward a hybrid approach that combined audio detection with visual facial metrics. By analyzing mouth shape and movement alongside sound, we were able to create a more reliable and interpretable measure of pronunciation accuracy.

2. Creating an Engaging UI
Another challenge was designing an interface that felt intuitive and motivating rather than overly technical. Simply displaying feedback or correction instructions was not enough to keep users engaged.

We addressed this by incorporating real-time visual cues directly onto the user’s face, along with a points-based system that rewards improvement speed. This transformed the experience from a corrective tool into an interactive game, making users more likely to continue practicing.

Accomplishments we're proud of

We’re proud to have built a technology product around one of the most human aspects of life: communication. Everyone has thoughts they want to share, yet uncertainty, insecurity, and difficulty with expression can prevent them from speaking up and being understood.

Over the past 24 hours, we found a new use for one of the world’s most engaging platforms. While Lens Studio (Snapchat) is often used for entertainment, we layered an educational experience on top of it.

Although there is still work to be done before reaching commercial viability, we strongly believe in the scalability and real-world impact of our product.

What we learned

Modeling is hard, and because of that, testing is everything. When building Verba’s accuracy system, we realized there are infinitely many ways someone can shape their mouth, adjust their tongue, and still correctly pronounce a word.

To account for this variability, we needed rigorous testing to ensure users weren’t incorrectly marked as wrong. As a result, the longest and most challenging part of this project was conducting hundreds of tests to determine acceptable ranges for key facial metrics.

In short, we learned that capturing human variation requires robust, large-scale data.

What's next for Verba

In the short term, we plan to:

Expand our word pool to cover a broader range of phonemes and difficulty levels. Due to the time of mapping out each word we were only able to build the tool for the word "dog"
Refine our facial metrics to identify more consistent and reliable indicators of pronunciation accuracy
Survey ESL learners to better understand potential hesitations and how users would engage with Verba, whether as a learning tool or a social experience

In the long term, we aim to:

Expand Verba to support multiple languages beyond English
Launch a social media campaign that encourages users to share their attempts, helping reduce the stigma and embarrassment often associated with learning to speak a new language