About the project — SignSpeak

Inspiration

I was inspired by how often communication becomes a barrier for Deaf and hard‑of‑hearing people in everyday situations—especially in high‑stakes environments like clinics, pharmacies, government offices, and classrooms. In many places, a professional interpreter is not always available at the moment it’s needed.
I wanted to build something practical and immediate: a tool that can translate sign language into spoken communication in real time, so that a Deaf user can be understood instantly—and also receive responses back in a sign-friendly way.

What it does

SignSpeak is a real-time communication bridge:

• Sign → Text → Voice: A user signs in front of a webcam. SignSpeak converts the signing into readable text and then speaks it out loud.
• Voice/Text → Sign (demo-friendly reverse mode): A hearing person can speak or type, and SignSpeak converts that into a sign-friendly output (via a controlled phrase library / sign animations for the MVP).

The core goal is simple: make conversations possible in seconds, not minutes.

What I learned

Building SignSpeak taught me a lot about:

• Multimodal AI design: translating visual input (webcam frames) into language requires careful prompt design, context handling, and stable output strategies.
• Real-time systems tradeoffs: frame rate, latency, and stability matter as much as model accuracy. In practice, “fast + consistent” beats “sometimes perfect.”
• Accessibility-first product thinking: the user experience should be calm, readable, and respectful—especially for medical or emergency contexts.
• Voice UX: using speech output can dramatically improve usability in real-world settings (e.g., a doctor doesn’t need to read a screen; they can hear it).

How I built it

I built SignSpeak as a small, modular system:

1) Frontend (Web)

• Uses WebRTC to capture webcam video
• Samples frames (e.g., every 0.5–1.0 seconds) and sends short bursts to the backend
• Displays live transcript and confidence/status
• Plays back generated audio (voice output)
• Provides a “reverse mode” UI (voice/text → sign)

2) Backend (API)

• A lightweight Python FastAPI server exposes endpoints like:
  • /translate/sign for sign → text
  • /tts for text → speech audio
• Handles basic session context, buffering, and response formatting.

3) Nova models on AWS

• Amazon Nova 2 Lite (via Amazon Bedrock) for multimodal reasoning:
  • Interprets the sign input from frame bursts
  • Produces a concise translation and optional confidence
• Amazon Nova 2 Sonic for voice:
  • Converts translated text into natural speech for the listener

4) Stabilization and reliability (important for live demos)

To reduce “text flickering” and improve consistency, I used:

• Temporal buffering: analyze a short window of frames instead of a single frame
• Output smoothing: only update the displayed translation when the model output is stable across consecutive windows
• Constrained phrase mode (MVP): for the hackathon version, I focused on a set of high-impact phrases (medical/emergency) to maximize reliability.

Challenges I faced

1) Sign language is not just hands

Sign language depends on motion, timing, and often facial expressions. Translating reliably from limited webcam frames is difficult. To manage scope, I designed the MVP around a supported phrase set and short phrase translation.

2) Real-time latency vs accuracy

If I send too many frames, the system gets slow; if I send too few, the translation can be unstable. I had to balance:

• latency (fast responses)
• stability (no flicker)
• accuracy (correct phrase)

A practical approach was sending a burst of frames representing a short time window and then smoothing results.

3) Demo robustness

Hackathon demos must work reliably. I focused on:

• predictable lighting/background suggestions
• clear UI cues (e.g., “Hold still”, “Sign again”)
• graceful fallbacks when confidence is low (returning UNCLEAR with prompts)

What’s next

After the hackathon, I want to expand SignSpeak toward:

• broader vocabulary and sentence-level translation
• better signer generalization (different people, camera qualities)
• richer reverse translation (more expressive sign animation / avatar)
• support for additional sign languages and community-driven phrase packs

Final note

SignSpeak is my attempt to use Amazon Nova to build something that’s not only technically impressive, but also meaningful—helping ensure that everyone can be heard, even when they communicate with their hands.

Built With

• amazon-s3-(optional:-storing-demo-media/logs)
• aws-lambda-+-api-gateway-(optional-deployment)-**apis-/-sdks:**-aws-sdk-for-python-(**boto3**)
• aws-lambda-+-api-gateway-(optional-deployment)-apis-/-sdks:-aws-sdk-for-python-(boto3)
• bedrock-runtime-api-**dev-/-tooling:**-github
• bedrock-runtime-api-dev-/-tooling:-github
• css-**frontend:**-react-(webrtc-webcam-capture)
• css-frontend:-react-(webrtc-webcam-capture)
• docker-(optional)
• html
• javascript/typescript
• languages:-python
• postman
• tailwind-css-**backend:**-fastapi-(python)
• tailwind-css-backend:-fastapi-(python)
• uvicorn-**ai-/-models:**-amazon-bedrock-with-**amazon-nova-2-lite**-(multimodal-reasoning)-and-**amazon-nova-2-sonic**-(speech)-**cloud-/-aws:**-aws-iam
• uvicorn-ai-/-models:-amazon-bedrock-with-amazon-nova-2-lite-(multimodal-reasoning)-and-amazon-nova-2-sonic-(speech)-cloud-/-aws:-aws-iam
• vite