Inspiration
Communication should feel natural for everyone. For many Deaf people who use American Sign Language (ASL), communicating with someone who doesn't know sign language often means switching to typing or relying on an interpreter.
This can become especially challenging in situations like telehealth appointments, where being able to express yourself clearly and independently matters.
We started SignFlow with a simple question: What if signing could feel as seamless as speaking into a voice-to-text app?
We wanted to explore how computer vision, AI, and desktop software could make communication more accessible.
What it does
SignFlow is a desktop application designed to turn recognized sign language gestures into English text and speech.
Our goal is to create a communication experience where users can sign in front of their webcam, review the recognized message, and choose to display, save, or speak it.
The application includes:
- Live Camera: A camera-first interface designed for sign recognition.
- AI-Assisted Text: A workflow for turning recognized sign tokens into readable English.
- Text-to-Speech: Users can confirm a message before having it spoken aloud.
- Floating Captions: A compact caption window that stays visible above other desktop applications.
- Writing Mode: Users can organize confirmed messages into notes and export them.
- Conversation History: Previously confirmed messages remain accessible during a session.
Our current desktop application supports these workflows using simulated recognition data while we integrate the sign recognition model and AI backend.
SignFlow is intended as an accessibility tool, not a replacement for qualified interpreters in critical medical situations.
How we built it
We divided SignFlow into three main components so each part could be developed independently.
1. Computer Vision and Sign Recognition
Our recognition pipeline is being developed using Python and computer vision techniques to identify supported ASL signs from webcam input.
2. AI Processing and Speech
Our backend is being developed with FastAPI, Pydantic-based data models, and AI services. The goal is to transform recognized sign tokens into understandable English while preserving their intended meaning.
3. Desktop Application
We built the desktop application using Electron, React, TypeScript, and Vite.
The interface includes a live webcam preview, an editable English draft, text-to-speech controls, notes, and an always-on-top caption window.
We also designed the frontend around typed adapters, allowing us to develop and test the entire desktop experience before connecting the real ML and backend services.
Challenges we ran into
One of our biggest challenges was understanding the difference between detecting hand movements and actually recognizing sign language.
ASL is a complete language with its own grammar, facial expressions, and movement patterns. Translating individual signs is not the same as understanding a continuous conversation.
Another challenge was building a desktop application that could access the camera, display floating captions, and communicate securely between Electron processes.
We also had to coordinate three independently developed components under a tight hackathon deadline. To make integration easier, we designed shared API contracts and prepared automatic TypeScript type generation from the backend's Pydantic/OpenAPI schemas.
Accomplishments that we're proud of
- Built a functional Electron desktop application with a camera-first experience.
- Implemented floating captions, editable drafts, speech controls, and note-taking functionality.
- Designed a modular architecture separating ML recognition, AI processing, and the frontend.
- Built automated tests for the desktop experience, including Electron end-to-end testing.
- Created a workflow that requires user confirmation before speaking generated text.
- Designed an integration approach that allows the three parts of the project to be developed in parallel.
What we learned
We learned that building accessible technology requires more than connecting an AI model to an interface.
Reliability, user control, and clear communication are especially important when the output represents another person's words.
We also gained experience with Electron security, camera permissions, computer vision integration, API contracts, and developing a complete desktop product as a small team.
What's next for SignFlow
Our next steps are to complete the integration between the sign recognition model, AI backend, and desktop application.
Longer term, we'd like to expand the supported vocabulary, improve continuous sign recognition, test with people who actually use ASL, and explore integration with communication platforms.
Our vision is to make SignFlow a practical tool that helps people communicate more independently, while being transparent about its limitations.
Built With
- amazonbedrock
- amazonpolly
- css
- electron
- fastapi
- html
- javascript
- langchaindeepagents
- mediapipe
- node.js
- playwright
- pydantic
- python
- react
- typescript
- vite
- vitest
