Inspiration
Multilingual people often engage in code-switching: alternating between two or more languages within a single conversation or even a single sentence. While natural, it can sometimes become an unconscious habit that makes verbal communication more difficult. Others may struggle with pronunciation due to strong accents, causing them to feel misunderstood or hesitant to speak.
Speech challenges can also affect neurodivergent individuals, whether through difficulty conveying emotions through tone, speech impediments such as stuttering or cluttering, or other communication differences that make conversations more demanding.
SpeakSense was built with these experiences in mind. By providing a safe and supportive environment for practice, the platform helps users improve their communication skills, build confidence, and feel more comfortable expressing themselves in everyday conversations.
What it does
SpeakSense offers users three different ways to practice and improve their speech:
Read Aloud Mode allows users to read text while the app analyzes pronunciation, stuttering, tone, and other speaking patterns in real time.
Conversation Mode lets users engage in natural conversations with an AI-powered chatbot, helping them practice speaking in realistic scenarios while tracking phonetic and fluency metrics.
Freestyle Mode generates prompts for users to respond to freely. The app can detect pauses, identify mistakes, suggest words when users struggle to find them—particularly useful for code-switchers—and provide improvisation feedback.
Across all modes, SpeakSense provides real-time audio visualization, tracking vocal frequency, speaking presence, and clarity metrics through interactive D3 visualizations. The app also generates transcript-based analyses to help users understand their speech patterns and monitor their progress over time.
All the user has to do is speak: SpeakSense captures audio through the microphone and delivers personalized feedback designed to improve communication skills and build confidence.
How we built it
Frontend Application (React/TypeScript):
- Provides three interactive speaking practice modes tailored to different communication challenges.
- Uses the Web Speech API to convert speech into text and generate spoken prompts from the AI coach.
- Uses the Web Audio API and D3.js to visualize speech in real time and track speaking activity.
- Stores user progress, streaks, and metrics locally for privacy.
Backend Server (Node.js/Express):
- Analyzes speech transcripts to identify filler words, code-switching, grammar issues, and speaking patterns.
- Powers the AI conversation mode by generating natural responses for speaking practice.
Tech Stack:
- React 18 + Vite
- TypeScript
- Node.js + Express
- Google Gemini API
- D3.js
- Tailwind CSS
- Web Speech API
- Web Audio API
Challenges we ran into
One of the main challenges encountered was the limitation of browser-based speech technologies. While they provide fast and accessible transcription, they can occasionally produce inaccuracies or hallucinations, particularly with accents, code-switching, or speech disfluencies.
Another challenge was AI integration. Ensuring that the chatbot delivered natural, high-quality conversations and generated realistic voice responses required significant iteration and fine-tuning.
We also placed a strong emphasis on UI/UX design. Since SpeakSense is intended to support users with diverse communication needs, we worked to create an interface that is intuitive, accessible, and comfortable for neurodivergent individuals.
Accomplishments that we're proud of
Designed an interface informed by research on dyslexia, sensory sensitivities, and diverse communication needs to make SpeakSense accessible and comfortable for a wide range of users.
Built a multilingual platform that supports users from different linguistic backgrounds, including those who frequently code-switch.
Successfully integrated speech recognition and AI-powered conversation features to provide meaningful speaking practice and feedback.
Created real-time audio visualizations and transcript analyses to help users better understand and improve their speaking patterns.
Developed a tool that empowers users to build confidence in their communication skills through regular practice.
What we learned
- Learned how to integrate speech recognition, audio processing, and AI models into a single application.
- Explored the challenges of working with speech technologies, including transcription inaccuracies and multilingual support.
- Improved our understanding of AI engineering, prompt design, and creating natural conversational experiences.
- Discovered the importance of balancing technical innovation with accessibility and user-centered design and gained experience designing accessible interfaces
What's next for Speak Sense
Improve the accuracy of speech transcription and AI feedback, particularly for accents, code-switching, and speech disfluencies.
Reduce response times and deliver feedback more quickly to create a smoother, more natural speaking experience.
Refine the conversational AI to provide more natural interactions and personalized coaching.
Expand language support to make SpeakSense accessible to users from a wider range of linguistic backgrounds.
Open-source the project to encourage community contributions and further research into accessible speech technologies.
Log in or sign up for Devpost to join the conversation.