YourTurn

What if every phone call waited for you?

YourTurn is an accessible call companion designed for people with speech or hearing disabilities. It helps users follow conversations, respond at their own pace, and stay in control of what gets communicated.

Our goal is simple:

Make the conversation adapt to the person — not force the person to adapt to the conversation.

---

Inspiration

Phone calls are built around one assumption:

You can hear, understand, think, and respond immediately.

For people who use AAC or have speech or hearing disabilities, that assumption can turn everyday calls — to a doctor, bank, employer, landlord, or even family member — into a barrier.

Captions help people understand what was said.

Text-to-speech helps people communicate a response.

But neither completely changes the fact that a traditional phone call moves at someone else's pace.

So we asked:

What if we redesigned the phone call itself?

Instead of expecting the user to keep up with the conversation, YourTurn gives them time to understand, prepare, edit, approve, and then communicate.

That became the foundation of our product:

AI can assist. The user decides.

---

What it does

YourTurn is designed to support the entire communication experience — before, during, and after a call.

During the call

YourTurn can provide:

• Live captions and transcription
• Caption and reply translation
• A clear focus on the caller's latest question
• Context-aware suggested replies
• The ability to select, edit, or completely rewrite a response
• Text-to-speech using a selected voice
• Controls to ask the caller to slow down, repeat, wait, or ask one question at a time
• Saved phrases for responses the user frequently uses

The most important rule is:

Nothing is spoken until the user approves it.

The AI may suggest a response, but the user always has the final say.

Before the call

A future version of YourTurn can help users prepare:

• The purpose of the call
• Important questions
• Facts they may need
• Things they want to remember
• Decisions they are or are not comfortable making
• Preferred introductions and communication instructions

After the call

YourTurn is designed to help identify important information such as:

• Dates
• Times
• Dollar amounts
• Appointments
• Commitments
• Decisions
• Next steps

The user can review this information before saving anything.

---

What we prototyped

For this hackathon, we focused on proving the most important part of the experience:

Can someone participate in a phone conversation without being forced to hear, process, and respond at someone else's pace?

We built an interactive browser prototype around a simulated healthcare call.

Users can:

• Follow captioned dialogue
• View sample Spanish translations
• Choose from suggested responses
• Edit a suggested response
• Type a completely new response
• Select a device voice
• Hear their response spoken only after pressing Speak reply
• Save frequently used phrases
• Ask for more time
• Request one question at a time
• Repeat the caller's last question
• Review key information after the call
• Provide feedback on the experience

Some elements of the demo — including the caller transcript, sample translations, and recap — are scripted so we could focus our hackathon time on validating the accessibility interaction and user-control model.

The prototype does not currently place or record a real phone call.

It is a proof of concept for the larger YourTurn experience.

---

Camera-based fingerspelling

We also experimented with another way for users to create a response.

Our prototype includes a camera-based ASL fingerspelling mode.

The current flow is:

Camera → Letter recognition → Editable text → User review → Confirmation → Spoken reply

The prototype recognizes a limited set of static fingerspelled letters and lets the user correct the interpretation before using it.

We intentionally do not describe this as full ASL translation.

ASL is a complete language involving movement, facial expression, grammar, spatial relationships, and cultural context.

Recognizing a static hand shape is very different from understanding ASL.

Any future expansion of this feature should be co-designed and tested directly with ASL users.

---

How we built it

We designed YourTurn around a human-in-the-loop communication model.

Instead of allowing AI to automatically speak for the user, our system is built around review and approval.

Core interaction

Caller speaks → Caption → User understands → Suggested reply → User edits → User approves → Text-to-speech

For our hackathon prototype, we used a scripted call flow to simulate the caller side while building the user interaction around it.

The prototype uses:

• HTML, CSS, and JavaScript for the interface
• Browser Speech Synthesis for spoken replies
• Device voice selection
• Local storage for saved quick responses
• A simulated captioned call workflow
• Sample translated captions
• TensorFlow.js for the fingerspelling classifier
• MediaPipe hand landmarks for hand detection and tracking
• A static ASL alphabet classification model
• Responsive and accessibility-focused interface design

For the production vision, the scripted portions could be replaced by:

• Real-time speech recognition
• Machine translation
• Language-model-assisted reply generation
• Information extraction
• Real phone or video-call integration

But our goal during the hackathon was not to connect the maximum number of APIs.

It was to prove the interaction model.

---

How we used Littlebird

One of the biggest challenges in a hackathon is not generating ideas.

It is remembering why decisions were made while the product is changing quickly.

We were constantly moving between:

• User research
• Product decisions
• Accessibility considerations
• Feature prioritization
• Prototype development
• Technical debugging
• Pitch preparation

Littlebird helped us keep that context connected.

Instead of repeatedly asking:

• What did we decide earlier?
• Why did we choose approval-before-speech?
• Which features belong in the prototype?
• What should stay in the future roadmap?
• What did we decide about ASL?
• What still needs to be completed?

we could continue working from the context that had already been created.

Why that was useful

With tools like ChatGPT or Claude, they were still extremely useful for brainstorming, reasoning, writing, and technical help.

But when starting a new conversation or switching tasks, we often had to rebuild context:

Explain → paste requirements → explain previous decisions → ask → continue

Littlebird helped shorten that workflow to something closer to:

Ask → continue

The biggest improvement was not simply how fast text appeared on the screen.

It was time-to-useful-answer.

During a short hackathon, avoiding repeated context reconstruction gave us more time to actually build and refine the product.

---

Challenges we ran into

1. Helping without taking control

The easiest version of YourTurn would let AI automatically generate and speak a response.

We deliberately chose not to do that.

If AI misunderstands the conversation and speaks something the user never intended, the accessibility tool has taken away the agency it was supposed to provide.

That led to our core principle:

AI proposes. The user decides.

---

2. Captions alone were not enough

At first, it was easy to think of the problem as a transcription problem.

But even perfect captions do not solve everything if the conversation continues faster than the user can respond.

That is why YourTurn also focuses on conversation pace.

Controls such as:

• Wait until I'm ready
• Ask one thing at a time
• Repeat the last question

are not secondary features.

They are part of making the conversation itself accessible.

---

3. Avoiding information overload

We could show:

• Full transcripts
• Translations
• Multiple suggestions
• Notes
• Controls
• AI analysis
• Call preparation information

all at once.

But an accessibility interface can become inaccessible if it overwhelms the user.

So we kept coming back to one question:

What does the user need to understand or do next?

---

4. Fingerspelling recognition

Computer vision can recognize some static hand positions.

That does not mean it understands ASL.

We deliberately reduced the scope of our prototype rather than overstating what the technology could do.

That became an important product lesson:

Responsible design sometimes means building less and being more accurate about what you built.

---

Accomplishments that we're proud of

We are proud that YourTurn became more than a collection of AI features.

It became a product with a clear philosophy.

Many AI products ask:

How much can we automate?

We kept asking:

How much control can we give back to the user?

Our prototype brings together:

• Captioned conversation
• Editable suggested replies
• User-written responses
• Text-to-speech
• Voice selection
• Translation examples
• Conversation pace controls
• Saved communication phrases
• Call recap
• User feedback
• Camera-based fingerspelling recognition

into one experience.

But one of our biggest accomplishments is actually something the product doesn't do.

It doesn't automatically speak for the user.

That limitation is intentional.

Because in communication, being fast is less important than being you.

---

What we learned

Our biggest lesson was:

Accessibility is not created simply by adding more AI.

A powerful AI system can still create a poor accessibility experience if the person using it loses control.

We learned that small product decisions can sometimes matter more than sophisticated models.

A button that says "Give me time" may be more valuable in a real conversation than another complicated AI capability.

We also learned the difference between AI capability and product trust.

An AI system may be capable of:

• Predicting someone's response
• Translating a sentence
• Recognizing a hand shape
• Extracting an appointment time

But being capable of making a prediction does not automatically mean the system should act on it.

For YourTurn, AI is most useful when it removes barriers between the user and the conversation — without becoming the person having the conversation.

---

What's next for YourTurn

The next step for YourTurn is not simply adding more features.

It is testing the product with the people it is intended to serve.

We want to co-design and test YourTurn with:

• AAC users
• People with speech disabilities
• Deaf and hard-of-hearing users
• ASL users
• Accessibility specialists
• Speech-language professionals

We would measure success through outcomes such as:

• Whether users can successfully complete important calls
• How long it takes to understand and answer a question
• Caption correction rates
• How often suggested replies need editing
• Accuracy of important dates and amounts
• Whether users feel in control
• Whether users can complete calls with less outside assistance

Future versions will include:

• Real-time speech recognition
• Real call integration
• Live translation
• Context-aware AI reply suggestions
• Call preparation tools
• User-approved communication memory
• Automatic detection of dates, amounts, and next steps
• Better personalization
• Stronger privacy controls
• Improved fingerspelling recognition
• Further sign-language research only through collaboration with sign-language users

---

The bigger vision

We were not trying to redesign captions.

We were redesigning the phone call around people the phone call was never designed for.

Traditional phone calls expect:

Hear → Understand immediately → Think immediately → Speak immediately

YourTurn changes that experience to:

Understand → Take your time → Choose → Review → Approve → Communicate

The goal is not for AI to speak for someone.

The goal is to make sure they get their turn to speak.

Your voice. Your pace. YourTurn.

Built With

• api
• app
• computer
• css3
• face
• getusermedia
• html5
• hugging
• javascript
• learning
• localstorage
• machine
• mediadevices
• mediapipe
• speech
• speechsynthesis
• storage
• tensorflow.js
• vision
• web