AccentFair — A Safety Layer for Voice AI
Inspiration
Voice assistants are becoming a part of everyday life, but speech recognition systems do not always understand everyone accurately. Differences in accents, pronunciation, and speaking styles can sometimes lead to incorrect transcriptions. While a small mistake might not matter when asking about the weather, it could become serious when giving financial instructions or other sensitive commands. We wanted to explore how AI systems could recognize uncertainty and involve the user before making a potentially harmful mistake.
What it does
AccentFair is a prototype safety layer for voice-based AI interactions. Users can record or upload audio, which is processed by two independent speech-to-text services. The application compares their transcriptions, highlights differences, and identifies potentially sensitive information such as amounts, dates, times, and account-related instructions.
Based on the results, AccentFair produces one of three decisions:
- PROCEED: The transcriptions agree and no sensitive instruction is detected.
- CONFIRM: The transcriptions agree, but the instruction contains potentially sensitive information.
- CLARIFY: The transcriptions disagree or a transcription service fails.
Users can review the transcriptions, replay their original audio, and verify an instruction before proceeding. The application also includes a Safety Playground with predefined and customizable scenarios to demonstrate its decision-making process.
How we built it
We developed AccentFair using Python and Streamlit. We integrated Groq Whisper and AssemblyAI as independent speech-to-text systems, used RapidFuzz and text normalization to compare their outputs, and built a rule-based risk engine to identify sensitive information.
We also implemented a human-confirmation workflow, transcript difference highlighting, audio replay, and an interactive testing playground.
Challenges
One major challenge was determining how to handle disagreement between AI models without assuming that agreement automatically means correctness. We also had to design confirmation logic that prevents outdated approvals from being reused when the audio or transcription changes.
Another challenge was combining multiple services into a simple interface while keeping the application testable without making repeated API requests.
Accomplishments
We built a functional prototype that processes real audio through two speech recognition services, explains transcription differences, and requests verification when appropriate.
We also created an interactive Safety Playground and developed an automated test suite with 340 passing tests.
What we learned
We learned that responsible AI is not only about improving model accuracy. It is also about designing systems that recognize uncertainty, communicate limitations, and give users control over consequential decisions.
What's next
Future improvements could include evaluating performance on public accent-diverse speech datasets, measuring error rates across speaker groups, expanding multilingual support, and integrating the safety layer with real voice-assistant workflows.
Limitations
AccentFair is a prototype and does not directly control Siri, Alexa, or financial systems. Agreement between two speech recognition services does not guarantee a correct transcription. We have not yet established that the system reduces accent-related performance disparities through a representative evaluation dataset.
Log in or sign up for Devpost to join the conversation.