TalkBuddy

Conversational assistant and courses to help people with autism

Comment

Inspiration

My brother has autism, and his experience (and the research) shows that people with autism can struggle to communicate in social situations. That can lead to being taken advantage of, having boundaries pushed, and not getting basic needs met.

We wanted to build something that supplements human support and gives people a safe place to practice the foundational skills of self-advocacy: asking for help, clearing up misunderstandings, setting boundaries, and expressing needs (like asking for a quieter space).

What it does

TalkBuddy is an AI-powered conversational partner that helps people with autism (or anyone who wants the practice) learn to advocate for themselves in social situations. The app uses several evidence-based methods supported by the National Clearinghouse on Autism Evidence and Practice's (NCAEP) 2020 review of methods. For example, it makes use of Social Stories, Prompting, and Reinforcement. It has two modes: Course Mode and Open Mode.

Course Mode

Four guided courses, each focused on one core skill:

  1. Asking for Help: requesting instructions from someone when you're stuck or confused.
  2. Setting Boundaries: communicating clearly when you're uncomfortable with something
  3. Expressing Needs: asking someone to accommodate for a problem you have, such as something being bad sensory.
  4. Clearing Up Misunderstandings: clarifying what you mean and understanding what the other person means.

Each course follows the same three steps:

  1. Learn: An introduction to the skill, with tips, encouragement, and things to focus on.
  2. Explore: An example scenario where the skill is needed, along with several response options. Users pick the one they like best.
  3. Practice: The heart of the course. Users role-play realistic scenarios with an AI conversational partner by speaking, typing, or using communication cards. They can end a session whenever they want and receive a reflection with a conversation summary, encouragement, and other things they could have said.

The courses are designed to incorporate several evidence-based practices for helping people with autism learn social and advocacy skills:

  1. Social stories: Each lesson presents a short scenario that explains what is happening, how the people involved might feel, and what a good response could look like.
  2. Prompting: When a learner is unsure how to respond, the app offers hints and cues that guide them toward an appropriate answer, so they can build confidence and rely on them less over time.
  3. Reinforcement: After each response, the app gives positive, encouraging feedback on what the learner did well, which motivates them to keep practicing.

Learn more about the efficacy of these practices here.

Open Mode

Users can enter any scenario they want to practice, beyond what the courses cover.

My Practice

Users can resume past conversations and review lessons learned whenever they're ready.

How we built it

  • Frontend: React
  • Backend & database: Supabase, for data storage and API calls
  • Conversation: Gemini API, which powers the AI conversational partner
  • Voice: ElevenLabs text-to-speech, which reads scenarios aloud and gives the AI partner a voice

Challenges we ran into

  1. Keeping conversations safe and on-track. Open-ended AI conversations can drift. We designed the courses around specific, common, important scenarios to keep practice focused and useful.
  2. Making text-to-speech feel natural. We didn't want users to have to take extra steps to hear the AI, so the conversational partner's replies are voiced automatically.
  3. Making the app comfortable for everyone. To accommodate for a vast range sensory needs, we made the app as customizable as possible. Users can make the app exactly what they want it to be: font, text size, line spacing, color themes, and voices can all be changed.

Accomplishments that we're proud of

  • User safety first. Given recent reports of LLMs wandering into unsafe territory, we built TalkBuddy to stop the conversation immediately if it veers into dangerous topics and point users to support resources.
  • Accessibility first. Users can interact with the partner in three ways: speech, typing, or communication cards (cards that signal meaning, like "I want to sleep" or "Stop"). With ElevenLabs TTS, they can experience the conversation through audio or text. We also followed W3C's Making Content Usable for People with Cognitive and Learning Disabilities guidelines so the app is easier to navigate and understand.
  • Customizable comfort. In User Preferences, users can change fonts (including OpenDyslexic), colors, text size, line height, line spacing, and the voice of the conversational partner. Customizable complexity. In the open practice mode, the user can choose between three different conversation difficulties: gentle, realistic, and challenging. Changing the difficulty doesn't make the conversation hostile, it only increases the complexity to mirror what they might experience in real life.

What we learned

We had to do a lot of research about how neurodivergent people interface so we could make the app as helpful and accessible as possible. This taught us a lot about the autistic experience.

This was our first time using the ElevenLabs TTS API, and we learned a lot about integrating it smoothly into the app. We also deepened our experience with the Gemini API and with connecting multiple services into one cohesive product. We had a lot of fun deciding coming up with the design for the app and implementing it.

What's next for TalkBuddy

  • More customization: additional voice characters and color themes
  • More communication cards, with helpful images
  • More scenarios and courses covering a wider range of real-life situations

Built With

Share this project:

Updates

Submission history