Hero

MANUS — Sign to Speech, Real Time

Inspiration

In the TV show "The Pitt," (HBO Max) the character Harlow Graham, a deaf patient, faces significant challenges in receiving timely and adequate medical care due to a lack of ASL interpreters and proper communication from staff.

What it does

MANUS is a sign-language-to-speech app for deaf and hard-of-hearing users who need to communicate with everyone else. Users open the app in the browser, grant camera access, and sign with their hands in view. The app captures video, sends frames to the backend, and uses the Gemini Live API to interpret sign language (e.g. American Sign Language) and speak the meaning in real time—so the other person hears natural speech while the user signs. No human interpreter required.

How I built it

Frontend: A single app page that requests camera permission, captures video at about 1 FPS, and sends frames to the server over a WebSocket. It receives streaming audio (and transcript) from the backend and plays the spoken interpretation. Binary WebSocket frames are handled so Gemini’s setup and server content decode and play correctly.
Backend: A Go HTTP server that serves the landing page, the app page, and a WebSocket proxy at /live. The proxy connects to the Gemini Live API (BidiGenerateContent), sends a session config with a sign-language-interpreter system instruction and response modality AUDIO, forwards client video as clientContent with turnComplete: true for immediate interpretation, and streams binary/text server content back to the browser.
Gemini Live: Uses the native-audio Gemini model so the model both interprets sign from video and responds with speech. The system instruction focuses on hands, face, and upper body and tells the model to output only the interpreted meaning—no chat or commentary.
Architecture: Configuration from .env (e.g. GEMINI_API_KEY, PORT). Docs in docs/APIS.md and docs/ARCHITECTURE.md describe routes, WebSocket flow, and data flow.

Challenges I ran into

Gemini Live protocol: The API sends setupComplete and some serverContent as binary WebSocket frames; the proxy had to handle both binary and text in the setup loop and when streaming interpretation so the frontend could decode and play audio.
Turn completion: Getting the model to respond immediately required sending video frames as clientContent with turnComplete: true so the model didn’t wait for end-of-turn from speech (which never comes in a sign-only flow).
Frontend binary handling: The browser needed ws.binaryType = 'blob' and correct handling of Blob/ArrayBuffer so that binary server content was decoded and played without errors.
[Add any other challenges: e.g. camera permissions, frame rate vs. latency, or model behavior.] ## Accomplishments that I'm proud of
A working real-time sign-to-speech pipeline in the browser with no installed app.
Keeping the Gemini API key on the server and exposing only a WebSocket proxy to the client.
Correctly handling Gemini Live’s mixed binary/text frames and native audio so interpretation plays reliably.
Clear docs (APIS, ARCHITECTURE) and a simple run story: go run . and open the app page.

What I learned

How the Gemini Live API works over WebSocket: setup message, client/server content, binary vs. text frames, and using AUDIO as the response modality for spoken output.
Designing a proxy that preserves streaming and binary content so the frontend stays simple and secure.
Trade-offs between capture frame rate, latency, and model behavior (e.g. when to send turnComplete) for a real-time sign-interpretation UX.
[Add what you learned about accessibility, sign language, or building with Go and WebSockets.]

What's next for MANUS

The voice-to-ASL use case is not available and probably more difficult to implement but not in scope for this Hackathon. I would like for MANUS to be 2-way conversation at some point.

Other options:

Support for more sign languages or locales.
Optional transcript display alongside audio for the conversation partner.
Tuning frame rate and prompts for lower latency and fewer missed signs.
Mobile-friendly UI and performance.
Offline or on-device option when such models become practical.

MANUS