Speech Odyssey

Inspiration

We wanted to create something that would help make healthcare accessible and low-cost, particularly for something like speech therapy that may be overlooked as a luxury in times of financial constraint. We were inspired by 1) another hackathon, where a participant used computer vision to develop an ASL-practicing tool, and 2) exposure to aphasia-related linguistics datasets in class

What it does

Our project aims to gamify speech therapy and provide a fun tool for practicing speech that makes the user more likely to be consistent and therefore see better results in the long term. We have fun text-based prompts (in the future text-based adventure game) that the user reads aloud. The user's audio input is captured and analysed on the basis on several relevant metrics used to monitor the progression of speech disorders and the effects of speech therapy. At the end of a session, the user can view how they did and compare their session's performance with historical data and see how they've improved over time.

How we built it

We used LLMs to generate our user's prompts, speech-to-text to transcribe their audio inputs, and text-to-speech to have an LLM say the same prompts so that the user has a baseline to compare their metrics against.

Challenges we ran into

Building the platform where the user does voice inputs; initially we were going to have a text-based adventure game but that proved complicated to implement. Chaining together multiple technologies was also difficult, as was finding reliable metrics in a domain we didn't have much expertise in.

Accomplishments that we're proud of

We're proud of whatever working parts we've built; this was our first time working with audio inputs and that was difficult to navigate but we managed it pretty well in the end. I also got my first exposure working with cloud technologies.

What we learned

To organise & plan better, coordinate better as a team, and not get hung up on the small details.

What's next for Speech Odyssey

Get domain expertise and improve our metrics & analysis. Make it a text-based adventure game. Create a working prototype with better UI. Implement a user profile so that the user can see their historical data and performance.

Built With

• html/css
• javascript
• python
• whisperapi