Inspiration

My goal was to develop something that would actually benefit individuals with disabilities. What about those who have no hand mobility at all? A lot of accessibility tools require some hand movement. HoyaAssist was designed to be entirely hands-free, requiring only voice and facial gestures for control.

What it does

HoyaAssist makes it possible to fully operate a computer without using your hands: You can move the cursor by moving your head. You can click automatically (dwell-click) by holding the pointer motionless for 1.2 seconds. To input text or provide instructions, use voice commands. Ask Gemini AI questions if you need help. ElevenLabs' text-to-speech feature allows you to hear responses. For a genuinely hands-free experience, everything comes together.

How we built it

There are three primary parts to the project:

Backend (FastAPI + Python) The Gemini 2.5 Flash API is integrated for AI answers. Connects to the ElevenLabs text-to-speech API

  • Responds to chat queries and serves audio files

Frontend (Tailwind CSS + React) Gorgeous UI with a purple/blue gradient; real-time chat with typing indicators Controls for audio playback Monitoring the status of connections

Controller (OpenCV + MediaPipe + Python) Using MediaPipe for face detection Controlling the cursor with head motions Dwell-click detection (clicking automatically) Using the Google Speech API for voice recognition Execution of keyboard commands (copy, paste, enter, etc.)

Challenges we ran into

Face tracking tuning: It required extensive testing to get smooth pointer movement. I repeatedly changed the screen coverage, smoothing parameters, and dead zones until it seemed natural.

Audio playback: Because the backend returned relative URLs, the frontend was unable to locate audio files. I had to create complete URLs that combined the audio path with the backend location.

Tailwind CSS configuration: Tailwind wasn't initially configured for the project. I modified the CSS imports, made configuration files, and manually installed Tailwind v3.

Gemini's 20 requests per day limit was reached during testing, necessitating the creation of a new API key and a backend restart.

Three-part integration: Careful design of the communication flow and error handling was necessary to get the controller, backend, and frontend to cooperate.

Achievements of which we are proud

created a hands-free solution that doesn't require a keyboard or mouse. Three distinct AI APIs (Gemini, ElevenLabs, Google Speech) were integrated; face tracking was used to create fluid, natural cursor control; a stunning, user-friendly web interface was created; and it was made modular so that each component could be built separately.

What we learned:

Using MediaPipe for real-time computer vision Creating user interfaces that are accessible to individuals with disabilities Combining several APIs with various needs Python threading and queue-based communication; contemporary React patterns with state management and hooks The significance of using user testing to adjust settings

What's next for HoyaAssist

Include a calibration screen for various camera configurations. Use cursor acceleration to move more quickly. Customize voice instructions for users; include eye tracking instead of face tracking; include session recording for accessibility studies; and develop an iOS/Android mobile version.

Built With

  • accessibility
  • ai
  • computer-vision
  • elevenlabs-api
  • face-tracking
  • fastapi
  • google-gemini-api
  • google-speech-api
  • javascript
  • mediapipe
  • opencv
  • pydantic
  • pynput
  • python
  • react
  • tailwindcss
  • uvicorn
  • vite
  • voice-control
Share this project:

Updates