Revoice

Listening to the speech.
Landing page to the ReVoice App.
Listening, understanding, and clarifying the words.
Tap speak and the app says your words aloud, clearly, once you've confirmed they're right.
One tap to start, with no scripts or setup.

Inspiration

Revoice started with a simple problem close to us.

One of our friends wanted to feel closer to his grandmother, but as her speech became more difficult to understand, even simple conversations could become frustrating. He wanted to listen and connect with her, but sometimes he simply couldn't understand what she was trying to say.

That made us think about how something as fundamental as speech can become a barrier between people. And this problem extends far beyond one family. People with speech impairments, including those affected by dysarthria, cerebral palsy, Parkinson's disease, stroke, and other conditions, can face the same challenge every day: they know what they want to say, but the technology and people around them may struggle to understand them.

This connected directly with the hackathon's theme, “Blocks that Make Up the World.” Communication is one of those fundamental blocks. It is how we build relationships, share ideas, ask for help, and stay connected with the people around us. When that block breaks down, people can become isolated.

So instead of asking people to change the way they speak, we asked:

What if technology could become better at understanding them?

That became Revoice. An AI-powered speech assistance app designed to make difficult-to-understand speech clearer while preserving what the speaker actually meant.

For us, Revoice is one small block that can help rebuild a much bigger one: human connection.

What it does

A user speaks naturally into Revoice. The app listens to their speech and uses speech recognition together with contextual speech repair to determine what they are most likely trying to say.

Rather than freely generating a replacement sentence, Revoice is designed to be conservative. When the system has strong evidence for a clearer interpretation, it can present or speak that interpretation. When it is uncertain, it tells the user instead of pretending to know.

The goal is simple: make the speaker easier to understand without changing what they meant.

How we built it

We built Revoice as an end-to-end mobile speech assistance system.

The mobile application was built with React Native, Expo, and TypeScript, with a FastAPI Python backend handling the speech-processing pipeline. When someone records their voice, the audio is sent to the backend and processed using Whisper for automatic speech recognition.

However, we quickly discovered that using a single Whisper transcription wasn't enough. When speech is difficult to recognise, the correct sentence may not be Whisper's first prediction, even though a better interpretation exists among its alternative hypotheses.

We therefore developed a hybrid N-best pipeline. Instead of considering only one transcription, Revoice generates several possible interpretations of the same speech. We then built and evaluated different approaches for choosing or repairing those hypotheses.

Our experiments included direct sequence-to-sequence speech repair, Flan-T5 contextual correction, and finally a constrained hypothesis-selection approach. The final direction deliberately restricts Revoice to selecting from evidence produced by the speech recogniser rather than freely inventing a replacement sentence.

We evaluated our approaches using dysarthric speech from the TORGO dataset with leave-one-speaker-out evaluation, so each test speaker was unseen by the model during training.

Finally, we connected the pipeline back to the mobile application so that the complete flow works from voice → recognition → interpretation → clear text → speech playback.

Challenges we ran into

The hardest problem wasn't simply improving transcription accuracy. It was improving accuracy without changing what the person actually meant.

Our first speech-repair model taught us this very quickly. A generative model could sometimes fix a badly recognised sentence, but it could also unnecessarily modify a sentence that Whisper had already recognised correctly.

We experimented with a larger Flan-T5 model and multiple speech-recognition hypotheses. This produced our strongest raw accuracy improvement, reducing word error rate from 31.75% to 27.23% in our evaluation. But the model could still occasionally generate a sentence that wasn't sufficiently supported by the original recognition hypotheses.

For an accessibility tool, we felt that was the wrong trade-off.

We therefore explored a constrained selector that can only choose from actual speech-recognition hypotheses. Our selected conservative system reduced word error rate from 31.75% to 28.49%, while producing zero unsupported generated sentences in our evaluation.

That experience changed how we thought about the project. The best model wasn't necessarily the one with the lowest error number. For Revoice, preserving the speaker's intent matters just as much as correcting their speech.

We also faced the practical engineering challenges of getting microphone recording, audio uploads, our Python backend, machine-learning pipeline, and playback working together on a real mobile application.

Accomplishments that we're proud of

We're proud that Revoice became more than a UI wrapped around an existing speech-to-text model.

We built and tested an end-to-end speech pipeline, investigated where conventional speech recognition struggles, and iterated through several different repair architectures before arriving at a safer approach.

One result we're particularly proud of is choosing not to use our most accurate generative model. Although it achieved a lower word error rate, it could occasionally produce unsupported corrections. We instead prioritised a constrained approach where every correction must come from evidence in the speech recogniser's candidate hypotheses.

Our final evaluated selector improved Whisper's baseline word error rate from 31.75% to 28.49%, with a 4.7:1 ratio of improved to worsened sentences, 77% edit precision, and zero unsupported generation in our evaluation.

Most importantly, we turned that research into a working mobile experience rather than leaving it as an offline machine-learning experiment.

What we learned

The biggest thing we learned is that building responsible AI isn't always about choosing the model with the highest headline accuracy.

Throughout the hackathon, our experiments repeatedly showed us a trade-off between aggressively correcting speech and preserving the speaker's original meaning.

We also learned that speech recognition contains more useful information than its single best transcription suggests. Looking at multiple hypotheses gave us alternative interpretations that were especially valuable when the first transcription was wrong.

Technically, we learned a lot about speech recognition, N-best decoding, contextual speech repair, model evaluation, confidence and uncertainty, mobile audio recording, and connecting a React Native application to a Python machine-learning backend.

But the most important lesson was much simpler: an accessibility tool should adapt to its user, rather than forcing the user to adapt to the technology.

What's next for Revoice

Revoice is currently a prototype, and there is still a lot we want to explore.

Our next priority would be personalisation. Speech patterns can vary significantly from person to person, so allowing Revoice to gradually adapt to an individual's speech could make recognition much more effective over time.

We would also like to improve how Revoice estimates uncertainty. Rather than treating every correction equally, future versions could better determine when a correction is safe to use automatically and when the speaker should be shown a few possible interpretations.

Another major direction is multilingual support. Our current prototype focuses on English, but difficulty being understood is not limited to one language. We want Revoice to eventually support speakers across different languages, accents, and linguistic backgrounds, making the technology accessible to a much wider range of communities.

Beyond that, we see Revoice working across more everyday communication settings, including face-to-face conversations, phone calls, and video calls, while supporting different types of speech difficulties.

Ultimately, we want Revoice to help remove communication barriers regardless of someone's speech, language, or age.

The long-term goal isn't to change someone's voice.

It's to remove one of the barriers preventing that voice from being understood.

Your voice, made clear.