Phonolens

see your speech

Comment

Inspiration

Growing up in a predominantly Asian community on Lenape land, heritage languages fill the community around me; Mandarin, Hindi, and a wide variety of other languages are spoken at dinner tables and on everyday walks. Yet the heritage language Munsee, the language of the land I live on, is clearly absent with only a select few elders speaking the language fluently today.

Discovering Munsee has forced me to confront the dangers of digitalization, especially considering connections and culture are easy to neglect. However, I hope to use technology to my advantage as a way to help preserve Munsee in my community through phonetics and build a platform which teaches others proper pronunciation of different languages.

What it does

PhonoLens is a pronunciation and intelligibility coach that treats your voice like data you can see. You record a target phrase, and the app renders the waveform, spectrogram, pitch contour, and formants side by side with a reference recording. It uses a dynamic time warping (DTW) alignment that highlights the specific time regions where your speech diverges, with A/B playback of just those slices; a live vowel-space chart that plots your F1/F2 formants in real time against the standard English vowel zones, so you can watch your vowel move as you adjust your tongue; minimal-pair drills (ship/sheep, bit/beat, code/cold) that give verdicts that judge vowel quality and vowel length as separate votes, plus final-consonant detection; and sagittal mouth diagrams that explain what your articulators are doing. It also includes an optional intelligibility check which asks a speech recognizer to transcribe you and scores it against the target, making sure to see if a machine understand the user. Additionally, PhonoLens makes sure that every measurement is computed on-device.

How we built it

The heart of the app is a speech-analysis library I built from scratch in TypeScript, it includes no external libraries. It breaks your recording into tiny slices and turns them into a spectrogram (a picture of your sound), tracks the pitch of your voice, finds your formants (the resonant frequencies that make an "ee" sound different from an "ah"), and lines your recording up against a reference to find exactly where they differ. It also trims silence and checks whether you actually pronounced the final consonant of a word.

Before building any of the interface, I tested the math with fake sounds where I already knew the right answer: I fed it a 220 Hz tone and checked it measured 220 Hz, and a computer-generated vowel to check it found the correct formants.

Challenges we ran into

The vowel detector started out confidently wrong. I tested it with a computer-generated vowel where I knew the correct answer, and it reported a frequency of 1212 Hz when the true answer was 2300 Hz; it was latching onto the wrong peak in the sound. I fixed it by switching to a more precise mathematical method for locating those peaks, and the test finally passed.

On top of that, silence almost broke comparisons since a recording might be 3 seconds long but contain only 1 second of actual speech, so the app was mostly comparing silence to silence. To combat this, I added automatic trimming to cut the quiet parts... which created a new bug: quiet word endings like the "p" in "sheep" or the "d" in "code" got trimmed off too — and one of our drills is literally about checking those endings. The trimmer had to learn to be aggressive at the start of a word but gentle at the end.

Detecting final consonants turned out to be tricky because every microphone picks up background hiss, so there is always some form of sound after the vowel. The detector only worked once it measured each recording's own background noise level first and required the consonant to clearly standout above it.

And I learned that the very beginning of a vowel is unreliable because the mouth is still moving into position, so the first few milliseconds give misleading measurements. Word drills (like ship vs. sheep) only became consistent when I measured just the stable middle portion of each vowel.

Accomplishments that we're proud of

I'm proud of creating a real-time, in-browser formant tracker that passes known-answer tests and correctly separates front and back vowels live. The drills are effective and honest. The app reports, in detail, what it measures (including telling you your vowel was too short even when its quality was right) instead of grading on a curve.

What we learned

From this, I have learned leaps about acoustic phonetics. More specifically, formants, vowel space, and why English final-consonant voicing lives mostly in vowel length. Additionally, I've learned about trimming voice clips based on certain criteria and comparing it to that of a reference voice clip. This serves as the basis for my project.

What's next for Phonolens

I'm hoping that PhonoLens will later be developed to include a larger database of words and languages. Specifically, I'm hoping to later include Munsee and work with local Princeton University professors to have PhonoLens serve as a tool for Munsee revitalization. When it comes to languages, they serve as a bridge between people to share culture. With the loss of language comes the permanent loss of centuries of culture and rich history. To preserve language is to preserve these past memories. and I hope PhonoLens can be used in the future to aid these efforts.

Built With

Share this project:

Updates