Architecture Diagram

Inspiration

I began learning Python four months ago and developed an initial version of this project, inspired by the physicist Stephen Hawking. Throughout the development process, I realized that a simple blink could become a means of communication. Thus, TalkinEyes OS was born, with the mission of ensuring that technology adapts to the user, rather than the other way around.

What does it do?

TalkinEyes OS is an assistive communication system that requires no manual input. It uses a standard webcam to detect facial landmarks and calculates the "Eye Aspect Ratio" (EAR) to identify intentional blinks. The user interacts with a selection menu (featuring categories such as "Food," "Restroom," "Pain," or "Sleep"). By blinking intentionally, the user selects an option, which the system then vocalizes using text-to-speech (TTS) technology. To ensure effectiveness in real-world scenarios, the system includes:

Personalized calibration: Learns the specific ocular geometry of each user.

Head-pose estimation: Pauses interaction if the user looks away.

Confidence layer: Ignores natural, rapid blinks to prevent accidental selections.

Predictive classification: Reorganizes the menu based on the options most frequently chosen by the user.

Emergency alerts: A simulated integration with AWS SNS to request urgent assistance.

How did I build it?

I used Python as the primary programming language.

Computer vision: I employed OpenCV and MediaPipe FaceMesh for real-time facial landmark detection, along with cv2.solvePnP to estimate the user's 3D head pose. Architecture: I evolved the project from a monolithic script (V1) into a modular, object-oriented (OOP) architecture. Functionalities were separated into distinct modules: core/ (vision), ui/ (Tkinter-based interface), calibration/, intelligence/ (local history management), and cloud/ (AWS integration).

Asynchronous Audio: I implemented a worker thread and a queue using pyttsx3, ensuring that speech synthesis does not block the Tkinter graphical user interface.

AI-Assisted Development: I used GitHub Copilot as a pair programmer to accelerate the writing of boilerplate code and assist in debugging complex mathematical APIs, allowing me to focus on software architecture and user experience (UX).

Challenges Encountered

OpenCV API Changes: The RQDecomp3x3 function—used to calculate Euler angles—returned data in a different format in the latest version of OpenCV. I consulted the documentation and adjusted the data-unpacking logic accordingly.

Memory Continuity: MediaPipe strictly requires arrays to be stored in contiguous memory blocks. I resolved a ValueError by using cv2.cvtColor instead of performing slicing operations on NumPy arrays.

Multithreading and SAPI on Windows: pyttsx3 was blocking the user interface, and the text-to-speech engine would stop producing sound after the first phrase. I resolved this by isolating the engine within a thread managed by a queue and re-instantiating the COM object for each phrase.

State Machine Logic: Preventing the system from selecting multiple options while the user’s eyes remained closed proved complex. I implemented a cooldown timer and a mechanism to reset the system state. ## Achievements I’m Proud Of I am quite proud of my "Adaptive Layer." It is more than just a blink detector: the system identifies who is blinking (calibration), where the user is looking (head orientation), how they blink (confidence level), and what they likely need (predictive classification). Furthermore, building a professional, modular architecture from scratch—at just 17 years old, with only 4–5 months of programming experience, and with the help of AI—is one of my greatest achievements.

What I Learned

I learned that writing code accounts for only 30% of software engineering. The real challenge lies in deeply considering architecture, state management, error handling, user experience, and all those small details that truly make a difference. I learned how to design classes using Object-Oriented Programming (OOP), how to manage concurrent threads in Python, and gained some insight into the underlying mathematical foundations of 3D matrices.s-OS