VakyaSetu

Inspiration

Watching a family member lose the ability to speak while their mind stayed completely sharp made one thing painfully clear: the barrier wasn't cognition, it was the interface between thought and the outside world. Existing assistive communication devices were either prohibitively expensive or clunky imports with no support for Indian languages. VakyaSetu (from "Vakya" sentence, and "Setu" bridge) was built to be that bridge: affordable, local language first, and fast enough for real conversation.

What it does

VakyaSetu is a low-cost assistive communication system for patients with paralysis who retain full cognitive ability. It pairs a custom Arduino-based hardware switch with a PyQt6 desktop application that gives the patient three ways to communicate:

Home screen a card carousel of common needs/phrases for one-switch, low-effort selection
VakyaKey a row/column scanning alphabet matrix with NLP-driven word suggestions, for composing free-form text one switch input at a time
Family chat an online chat feature so the patient can message family members directly from the app

The whole system supports both Hindi and English, with audio output via pygame so typed messages can be spoken aloud.

How we built it

Hardware: Arduino based switch input, communicating with the desktop app over serial
Software: PyQt6 for the desktop UI, pygame for audio playback, custom NLP logic to drive word suggestions in VakyaKey
Localization: Hindi/English string handling built in from the start rather than bolted on
Cost: the entire working prototype was built for under ₹1000, keeping it realistic for the patients who need it most

Challenges we ran into

Threading: keeping the serial-read loop from blocking or racing with the PyQt6 UI thread
Serial port lifecycle: reliably opening, holding, and releasing the Arduino connection across app states without hangs or silent disconnects
UI consistency: style bugs surfacing across the different screens (home, VakyaKey, chat)
Import-time errors: dependency and module-load issues that only showed up under certain run conditions
Scanning speed vs. accuracy: tuning the row/column scan rate in VakyaKey so it was fast enough to be usable but not so fast that an intended selection got missed

Accomplishments that we're proud of

A fully working assistive communication loop hardware switch to spoken sentence for under ₹1000 in parts
Three distinct, purpose-built ways to communicate (quick-select, free typing, real-time chat) instead of forcing every use case through one interface
Native Hindi/English support, not an afterthought
Catching and fixing serious reliability issues (threading, serial lifecycle) through detailed code review before they became patient-facing failures

What we learned

Assistive tech has to be designed around the constraints of the input method first everything else (UI, features) has to bend around what a single, effortful switch press can reliably do
Serial communication with hardware exposes UI threading bugs that don't show up in normal software reliability work here isn't optional polish, it's core to whether the device is usable
Cost discipline is itself a design constraint, not just a budget line keeping the build under ₹1000 shaped hardware choices from day one
Local language support changes adoption more than any single feature does

What's next for VakyaSetu

Integrating a local, on-device personalization agent a small n-gram language model trained on each patient's own typing history to improve prediction speed across the home screen, VakyaKey, and chat
Expanding language support beyond Hindi/English
Hardening the serial connection and threading model further for long duration, unattended use
Field testing with more patients to refine scan speed and switch sensitivity per individual need