Sai Baba Hackathon group

Inspiration

The question of "Build What's Next" kept making us go back to the same question: what exactly does "next" mean for someone who doesn't have any way to ask for it? People living with conditions like ALS, cerebral palsy, or the aftermath of a devastating stroke can have a fully intact mind behind a body that can move only a finger. The devices that currently exist to give their voice back are often rigs costing thousands of dollars, which are tuned by specialists, speaking in the same flat voice no matter who the user is behind the rig.

What it does

Whisper pairs an ESP32 with an AI agent that speaks on someone's behalf. The device reads the whole room around the user, accounting for factors including temperature, humidity, and ambient light, and takes one tiny physical input, sized to whatever movement the person makes, in this case, a thumb on a joystick, a head tilt tomorrow. That environmental and physical context, with the time of day and the person's recent history, gets sent to an AI agent that predicts a list of things they're likely trying to say out loud. One small nudge moves through the options, one press selects, and the agent expands that choice into a full sentence based solely on the context of the situation, which is spoken aloud in real time. If someone chooses the wrong option, a single movement rejects it and asks again. And when the system remembers, it's closer to being right the next time.

How we built it

ESP32 reads the joystick plus a DHT11, which measures temperature and humidity, and a photoresistor, posting each event to the backend through WiFi. FastAPI runs the whole thing as a live state machine, starting from idle - predicting - confirming - speaking, which pushes every single state change to the frontend over a WebSocket the instant input is received. Turso stores every phrase Whisper had ever proposed, rejected, selected, or spoken. Both the memory that personalizes future predictions and the receipts prove that the system is actually learning. A 3D-printed mount holds the input exactly where the user's one small movement lands, because assistive hardware only works if it's shaped around the body, not the other way around. We used a pre-existing Ollama model and then paired it with a GPT-4.1 mini in order to achieve the best and quickest results.

Challenges we ran into

A big challenge was integrating hardware, as that was the core of our project, but syncing and routing with the backend took a lot of time and energy, almost draining us completely. Another issue was training the AI model, as we had never done it before.

Accomplishments that we're proud of

We are proud of being able to integrate and push AI into a field that needs changes. Products these days are very overpriced for no reason, and aren't that great in terms of quality. We are very happy to be solving an issue that needed a solution.

What we learned

Do not do hardware projects if you do not prepare for hardware. We did not prepare for hardware before and just decided to do it in person.

What's next for Sai Baba Hackathon group

We want to implement a few features that would elevate our project, which would take some more time, but we could definitely do it. The ideas we would want to do are eye tracking and also using google search for a bigger database of sentences and words.