LipLingo

Inspiration: Millions of people around the world live with hearing or speech impairments, yet most of our everyday communication tools are built exclusively for those who can hear and speak. We watched videos of deaf individuals struggling to communicate in hospitals, classrooms, and workplaces — places where clear communication can literally save lives. We asked ourselves: what if a smartphone could bridge that gap instantly? That question became LipLingo. We wanted to build something that didn't just look impressive in a demo, but genuinely made a difference in someone's daily life.

What it does LipLingo is a dual-mode real-time communication accessibility app designed for deaf, hard-of-hearing, and speech-impaired individuals. Listen Mode — Point your phone camera at anyone speaking. LipLingo captures their speech via microphone, transcribes it instantly into large, bold text on screen, and translates it into your preferred language from 20+ supported world languages including Hindi, Tamil, Spanish, French, Arabic, Mandarin, and more. Speak Mode — For speech-impaired users who cannot speak, type what you want to say or tap from a customizable quick-phrase board. LipLingo displays your message in massive high-contrast text for others to read, or speaks it aloud using text-to-speech in your chosen language. Additional features include full conversation history, session export, an emergency SOS button that sends a pre-written message to a saved contact, dark mode, high-contrast accessibility settings, and adjustable font sizes.

How we built it We built LipLingo entirely using MeDo, having detailed multi-turn conversations to design, build, and refine each feature step by step.

Speech transcription — Whisper API for accurate real-time speech-to-text across multiple languages Translation — Google Translate API with auto language detection to convert transcribed text into the user's preferred language Text-to-speech — Web Speech SynthesisAPI built into the browser, requiring no external service and working across all devices Camera integration — Live camera viewfinder with face detection overlay in Listen Mode, giving users a natural face-to-face communication experience UI/UX — Accessibility-first design with large fonts, high contrast, and minimal navigation built for users with disabilities

We structured our MeDo conversations by tackling one feature at a time — first the core transcription, then translation, then TTS, then camera, and finally accessibility polish.

Challenges we ran into Translation API failures — Our first attempts using MyMemory API returned incorrect language codes and silent failures. We iterated multiple times before finding a reliable solution with proper ISO language code mapping and auto language detection. Text-to-speech compatibility — External TTS APIs kept failing, so we switched to the browser's built-in Web Speech Synthesis API which proved far more reliable across devices. Lip reading limitations — Our original vision included real-time lip reading via camera. Through research and MeDo's guidance, we learned that no publicly available API can reliably deliver this yet. We pivoted to a camera-plus-microphone approach that looks and feels natural while delivering accurate results — a better experience for our actual users. Accessibility balance — Designing for users with disabilities requires extra care. Getting the font sizes, contrast ratios, and navigation simplicity right took several iterations.

Accomplishments that we're proud of Built a fully working accessibility app that serves two distinct user groups in one clean interface Successfully integrated real-time speech transcription, multi-language translation, and text-to-speech without any paid API keys Designed an emergency SOS feature that could genuinely help a deaf or mute person in a crisis situation Created an experience that feels natural — watching someone speak while their words appear on screen is powerful and intuitive Made every design decision with accessibility as the priority, not an afterthought

What we learned Real-world accessibility needs are far more nuanced than they first appear — what works for a hearing user often fails completely for a deaf one MeDo is remarkably powerful for rapid prototyping — having a conversation to build features is faster and more intuitive than writing code from scratch Reliability matters more than ambition — a feature that works 100% of the time beats a flashy feature that works 60% of the time, especially for users who depend on it Free APIs like Web Speech Synthesis can outperform paid alternatives when used correctly Iterating quickly based on real testing feedback is the most effective development approach

What's next for LipLingo True lip reading — As AI vision models become publicly accessible, we plan to integrate genuine lip movement analysis to fulfil our original vision Sign language detection — Computer vision to recognize ASL, BSL, and ISL hand signs and convert them to text in real time Offline mode — Full functionality without internet for users in low-connectivity areas Wearable integration — Smartwatch companion app that vibrates and displays transcribed text on the wrist Healthcare mode — Specialized quick-phrase boards for hospital and medical settings Community phrase library — Let users share and download custom quick-phrase boards for specific situations like travel, school, or work