Inspiration
We both practise public speaking, and the feedback apps we tried kept saying things like "you sound monotone" or "slow down". That never told us which words were the problem or what to change. We wanted a tool that points at the exact words, shows the numbers behind its opinion, and can check whether fixing those words actually helps.
What it does
You read a short text out loud. ProsoDiff compares your reading with a few good readings of the same text, one word at a time.
For each word it measures ten things, such as how fast you said it, how long you paused after it, how much your pitch moved and how loud it was. It then checks how far each number is from what the good readers did:
$$z = \frac{x - m}{s}$$
Here $x$ is your value, $m$ is the typical value among the good readings and $s$ is how much good readers normally vary. These are combined into one score per word, $D = \sum_f w_f |z_f|$. When $D$ stays high over a few words, ProsoDiff marks that part and says what kind of problem it looks like: rushing, dragging, dead air, missing pause, monotone or trailing off. It also explains why, for example "you paused 2.8 s here, the good readings pause 0.03 s".
We also wanted a way to test the diagnosis itself, so we added "Prove the Fix". ProsoDiff edits your own recording, changing only the flagged words and only the thing it blamed, and analyses it again. If the flag goes away, the diagnosis was probably right. As a check it also changes something it did not blame, and that change should leave the flag where it is.
There is no AI model writing the feedback. Every sentence is filled in from the measurements.
How we built it
We used four presidential speeches (Nixon, FDR, JFK and Reagan) because U.S. government recordings are public domain and we are allowed to edit them. We added 16 computer-generated readings of the same texts as extra good examples.
To have something to test on, we took good recordings and added mistakes on purpose at four strengths. Since we made the edits ourselves, we know exactly where each mistake starts and ends. That gave us 260 recordings, about 3.6 hours.
The backend is Python. A speech alignment model finds when every word starts and ends, and Praat measures pitch and loudness. The website is React. It shows your voice next to the good readings on charts that move together, lists the flagged parts with explanations, and has the Prove the Fix button. We kept one speech and one voice hidden the whole time and only tested on them at the end.
Challenges we ran into
The alignment model gave very short words almost no length; it thought "I" lasted 0.02 seconds. We fixed it by also looking at loudness to see where each word really ends.
Praat turned out to be slightly random. The same input gave slightly different audio each time, which we only noticed because we ran everything twice and compared. Setting a fixed random seed solved it.
Real data was messier than we expected. "1941" has to be read as "nineteen forty-one", "Mr." as "mister", the FDR recording is from 1941, and JFK was speaking outdoors to a crowd.
The hardest problem is still open. Most of our good readings are computer voices, which are flatter and pause less than real speakers. So when a real speaker goes flat, they end up closer to the computer voices, and ProsoDiff finds it hard to notice.
Accomplishments that we're proud of
We built a dataset where every mistake has a known position, and we checked the labels again independently. 81% were accurate to within 0.05 seconds, and we marked the rest instead of hiding them.
On the hidden test set, every flag ProsoDiff raised was a real mistake, with no false alarms on 38 clean recordings. It does miss a lot, though. It found 45% of the dead-air mistakes and almost none of the other types. We decided on our settings before testing and did not change them afterwards, so these are the real numbers.
Prove the Fix worked on a hidden Reagan recording. Fixing the pause it blamed dropped the score from 2.86 to 1.25 and the flag disappeared, while changing pitch instead left it at 2.94.
What we learned
Tests passing did not mean the output was right. All our tests passed while real speech still had the 0.02-second word bug, so now we look at actual results as well.
Keeping the test data completely separate was harder than it sounds, but it is the only reason we trust our numbers.
Averaging many measurements can hide one big problem. One 3.5-second silence was extremely unusual on its own ($z = +15.7$), but averaged with nine normal measurements the score stayed just under our alarm line ($D = 2.80$ against a threshold of $3.0$).
What's next for ProsoDiff
The most important step is recording real people reading the texts well, so ProsoDiff compares you with humans instead of computer voices. We also want a second rule that raises a flag when any single measurement is extreme, and we want two people to label some recordings by ear so we can check our labels against human judgement. After that, we would like to make the analysis faster, since it currently takes around 30 seconds per recording.
Log in or sign up for Devpost to join the conversation.