SpeakEasy

Inspiration

We were motivated by the communication barriers between sign language users and non-signers. Many existing tools are one-directional or lack accessibility, and we wanted to create a mobile-first solution that empowers smoother, more inclusive interaction using everyday devices.

What it does

It captures spoken language and converts it into 2D ASL animations.
It also captures ASL hand gestures via camera, classifies them using hand landmarks, and outputs spoken language using text-to-speech.
The app processes input step-by-step, allowing users to control the flow of conversation.

How we built it

Frontend: React Native with Expo for cross-platform support
Backend: Python Flask API hosted on AWS EC2
Gesture Recognition: MediaPipe Hands to extract real-time hand landmarks
ASL Classification: Custom model trained on ASL alphabet data
Speech Synthesis: Google Gemini API for lifelike text-to-speech
ASL Output: 2D animations rendered on the mobile frontend

Challenges we ran into

Designing a clean, accessible UX for a two-step interaction flow
Ensuring gesture detection accuracy in varied lighting and backgrounds
Managing latency and performance across the mobile–server–AI pipeline
Integrating and syncing multiple services (MediaPipe, Gemini API, Flask server)

Accomplishments that we're proud of

Successfully built a working prototype that handles both directions of communication
Integrated ASL detection with 2D animations and natural speech output
Created a mobile-first interface that’s simple, intuitive, and accessible
Achieved reliable gesture recognition with minimal training data