Inspiration Sowtee was inspired by people who have ideas, humor, and identity—but lose speed of expression due to ALS/MS and related conditions. Stephen Hawking’s story showed us the same gap at global scale: intelligence is intact, but communication bandwidth is painfully low. We wanted to build a system that doesn’t just assist speech, but learns the person behind it.
What it does Sowtee is a self-learning communication agent for the voiceless. It takes eye-gaze input, predicts likely phrases, lets users accept/edit quickly, and speaks naturally using ElevenLabs. With every interaction, it adapts suggestions to the user’s habits, vocabulary, and context.
How we built it We built a multimodal stack with a Nuxt/Vue frontend, FastAPI backend, and a learning loop around user feedback. AWS Bedrock powers reasoning, prediction, and context understanding. ElevenLabs handles expressive text-to-speech. A memory and metrics layer tracks profile signals, acceptance rate, latency, and vocabulary growth to improve future suggestions.
Challenges we ran into Real-time latency under live demo constraints, balancing aggressive prediction with user trust, and making feedback-driven learning stable (without noisy over-corrections) were our biggest technical hurdles. We also had to design for clarity: reducing clicks while keeping full user control.
Accomplishments that we're proud of We delivered a working end-to-end communication flow: eye-gaze input → prediction → selection/edit → natural speech → learning update. We integrated AWS Bedrock and ElevenLabs effectively, and proved measurable in-session improvement through acceptance and prediction metrics.
What we learned Assistive AI success is not just model quality—it’s speed, personalization, and dignity. Small feedback signals (accept/reject/edit) are powerful for adaptation. Human-centered UX decisions often matter as much as algorithmic improvements.
What's next for Sowtee Next, we plan to expand language coverage, strengthen personalization over long-term use, run pilot testing with clinics/caregivers, and add stronger outcome tracking (fatigue reduction, communication speed, and quality-of-life signals). The goal is to move from a strong demo to real daily assistive impact.
Built With
- adaptive-suggestion-engine
- css-frontend-frameworks:-nuxt-3
- docker-compose
- elevenlabs-api-(text-to-speech)-databases-and-storage:-chroma-vector-database-(with-sqlite-persistence)
- html
- javascript
- languages:-python
- local-linux-runtime-scripts-for-one-command-demo-startup-core-capabilities-integrated:-eye-gaze-input-flow
- memory/personalization-loop
- multilingual
- prediction
- profile-and-learning-json-logs-platform-and-tooling:-docker
- support
- tailwind-css-backend-framework:-fastapi-ai-and-cloud-services:-aws-bedrock-(reasoning
- translation)
- typescript
- vue-3