CoSign

Inspiration

Video calls assume everyone can participate through speech. For Deaf people whose first language is ASL, that can mean typing while the conversation moves on or relying on an interpreter.

We reached out to more than 20 ASL college clubs and Deaf faculty across the U.S. Feedback from Deaf ASL faculty at the University of Washington highlighted accuracy, privacy, and the importance of involving Deaf people in the design process.

That shaped CoSign’s goal: give signers more control over both the conversation and what happens next.

What it does

CoSign is a native Mac app that helps Deaf ASL signers participate in Google Meet calls:

Sign to speak: turns supported ASL signs into spoken English.
Read the conversation: captions incoming speech and highlights mentions of the signer’s name.
Spell less: learns names and jargon, then suggests complete words from fingerspelled prefixes.
Turn discussion into action: drafts Jira tasks from meeting commitments and waits for the signer’s approval before requesting creation.
Stay in control: offers optional translation review, quick replies, and conversation-aware speech timing.

Our prototype supports a limited sign vocabulary and held fingerspelled letters. It is an early accessibility tool, not a replacement for qualified interpreters.

How we built it

We connected a SwiftUI interface to a Python asyncio engine through local JSON messages.

Recognize: MediaPipe extracts webcam landmarks; local models and custom recordings identify supported signs and letters.
Translate: Claude converts recognized ASL gloss into natural English.
Speak and caption: Cartesia provides speech generation, incoming transcription, and turn detection; BlackHole routes audio into and out of Meet.
Remember: SQLite stores vocabulary locally, and Moss retrieves contextually relevant completions.
Act: a Claude-powered assistant drafts meeting tasks, with a Forge bridge for approved Jira creation.

Privacy boundaries: camera recognition runs locally. Translation and topic tagging send text to Claude; speech and captions use Cartesia. Remembered vocabulary stays in a local database that users can inspect and delete.

How we used sponsor technologies

Anthropic — community-focused AI: Claude translates ASL gloss into spoken English, generates topic tags for autocomplete, and drafts meeting tasks. These functions serve a specific community, with optional speech review and required task approval to preserve signer control.
Cartesia — voice at the core: Sonic streams the signer’s English into Meet, while Ink provides live captions, name/jargon keyterms, and turn detection. Voice and transcription power the main communication loop, including timing speech around other participants.
Moss — contextual memory: a Moss session index searches locally remembered names and jargon using conversation context. We combine its relevance scores with letter matching, frequency, and recency so autocomplete does more than match a prefix.
Atlassian — useful work with approval: our Jira integration drafts tasks from meeting commitments and includes an authenticated Forge bridge for creation. The signer approves the current draft before a creation request is sent, and the request includes approval/version metadata for an audit trail.

Challenges we ran into

Recognition accuracy: hand shape, movement, timing, and individual signing styles affect meaning. We constrained the demo vocabulary, recorded custom signs, and stabilized predictions to reduce accidental triggers.
Audio routing: incoming call audio and generated speech need separate virtual devices so CoSign does not transcribe itself.
Speed and control: translation, autocomplete, and task suggestions must remain responsive without taking decisions away from the signer.
Honest interpretation: fluent English is not proof of accurate recognition. We keep the prototype’s limitations visible and distinguish intended meaning from model confidence.

Accomplishments that we're proud of

Connecting local recognition, translation, speech, captions, and contextual memory in one native interface.
Turning a name heard during a meeting into a useful fingerspelling suggestion.
Implementing a meeting-to-task workflow with explicit signer approval.
Grounding our product direction in Deaf educator feedback rather than relying only on our assumptions as a hearing team.

What we learned

Accessibility is about participation, timing, trust, and agency. Converting signs into English is only one part of helping someone participate in a meeting.

We learned to separate recognition confidence from preserved meaning, design around user control, and treat continued collaboration with Deaf signers as essential to evaluating the product.

What's next for CoSign

Test and refine the experience with Deaf ASL signers.
Expand sign vocabulary and improve fingerspelling speed and accuracy.
Validate complete two-laptop calls and measure real-world latency and recognition quality.
Strengthen privacy controls and refine approval interactions.
Improve Jira context handling to suggest updates to existing work and reduce duplicate tasks.