PatientEase

My project is PatientEase Pro, a hands-free navigation system for patients with motor disabilities. It allows users to control an entire web browser, scroll pages, click buttons, and search the web using only head movements, eye gaze, and natural voice commands—completely eliminating the need for a mouse or keyboard.

Also the split screen that you see is basically frontend+backend....the left side shows the frontend UI from where you will access and control the website by enabling voice commands and webcam and in the right side....it is the backend opening the website you gave command and you will see changes there.

CORE ARCHITECTURE & TECH STACK (How it works) Macro-Movements (Scrolling): We use OpenCV’s Haar Cascade Model to track the user's face. By calculating vertical displacement changes in real-time, it instantly detects head tilts to trigger upward or downward page scrolls.

Micro-Movements (Precision Clicking): We implemented MediaPipe’s Face Mesh API to track the coordinates of both irises. Once calibrated, a steady dwell/fixation on any screen point registers a continuous progress buffer that executes an auto-click.

Browser Execution Engine: The backend is powered by FastAPI and Selenium Automation. When a face gesture or voice request is processed, FastAPI injects optimized local scripts into an automated Chrome driver for instant web interactions, while PyAutoGUI handles background native OS interactions.

Native Voice Processing: We use the browser's built-in Web Speech API on the Streamlit frontend. It streams local microphone audio into high-fidelity text blocks instantly without adding system overhead.

Intelligent AI Routing: The transcribed text is sent to our core routing brain backed by Google Gemini 3.5 Flash. Gemini acts as an intent mapper: it translates natural commands like "open shopping portal" into absolute URLs (like Amazon) or packages general questions into formatted query parameters to navigate the patient straight to their destination.