Inspiration
My goal was to develop something that would actually benefit individuals with disabilities. What about those who have no hand mobility at all? A lot of accessibility tools require some hand movement. HoyaAssist was designed to be entirely hands-free, requiring only voice and facial gestures for control.
What it does
HoyaAssist makes it possible to fully operate a computer without using your hands: You can move the cursor by moving your head. You can click automatically (dwell-click) by holding the pointer motionless for 1.2 seconds. To input text or provide instructions, use voice commands. Ask Gemini AI questions if you need help. ElevenLabs' text-to-speech feature allows you to hear responses. For a genuinely hands-free experience, everything comes together.
How we built it
There are three primary parts to the project:
Backend (FastAPI + Python) The Gemini 2.5 Flash API is integrated for AI answers. Connects to the ElevenLabs text-to-speech API
- Responds to chat queries and serves audio files
Frontend (Tailwind CSS + React) Gorgeous UI with a purple/blue gradient; real-time chat with typing indicators Controls for audio playback Monitoring the status of connections
Controller (OpenCV + MediaPipe + Python) Using MediaPipe for face detection Controlling the cursor with head motions Dwell-click detection (clicking automatically) Using the Google Speech API for voice recognition Execution of keyboard commands (copy, paste, enter, etc.)
Challenges we ran into
Face tracking tuning: It required extensive testing to get smooth pointer movement. I repeatedly changed the screen coverage, smoothing parameters, and dead zones until it seemed natural.
Audio playback: Because the backend returned relative URLs, the frontend was unable to locate audio files. I had to create complete URLs that combined the audio path with the backend location.
Tailwind CSS configuration: Tailwind wasn't initially configured for the project. I modified the CSS imports, made configuration files, and manually installed Tailwind v3.
Gemini's 20 requests per day limit was reached during testing, necessitating the creation of a new API key and a backend restart.
Three-part integration: Careful design of the communication flow and error handling was necessary to get the controller, backend, and frontend to cooperate.
Achievements of which we are proud
created a hands-free solution that doesn't require a keyboard or mouse. Three distinct AI APIs (Gemini, ElevenLabs, Google Speech) were integrated; face tracking was used to create fluid, natural cursor control; a stunning, user-friendly web interface was created; and it was made modular so that each component could be built separately.
What we learned:
Using MediaPipe for real-time computer vision Creating user interfaces that are accessible to individuals with disabilities Combining several APIs with various needs Python threading and queue-based communication; contemporary React patterns with state management and hooks The significance of using user testing to adjust settings
What's next for HoyaAssist
Include a calibration screen for various camera configurations. Use cursor acceleration to move more quickly. Customize voice instructions for users; include eye tracking instead of face tracking; include session recording for accessibility studies; and develop an iOS/Android mobile version.
Built With
- accessibility
- ai
- computer-vision
- elevenlabs-api
- face-tracking
- fastapi
- google-gemini-api
- google-speech-api
- javascript
- mediapipe
- opencv
- pydantic
- pynput
- python
- react
- tailwindcss
- uvicorn
- vite
- voice-control