TalkTuahWebsite

TalkTuahWebsite
System Architecture
Problem Statement
Workflow

Inspiration

In the United States, approximately 20 million Americans live with some form of visual disability. Additionally, an estimated 88% of websites remain inaccessible, further limiting access to information and services. Many individuals with some form of visual impairment who could benefit from screen readers find existing options too complex or rigid, resulting in them not being able to engage fully with their devices and the web. Other potential users, like those with dyslexia (15 million Americans), also serve to benefit from personalized accessibility tools that can make interacting with content more intuitive.

These statistics underscore the urgent demand for a more user-friendly, intuitive approach to bridge the gap between accessibility challenges and our digital platforms.

What it does

TalkTuahWebsite is a personalized, conversational web assistant that leverages multimodal inputs and real-time capabilities to provide a more interactive, and user-friendly experience for individuals with visual impairments and neurodivergent conditions.

Our system enables users to navigate digital contact through natural, voice-based interactions, offering personalized assistance and seamless use across various sites and platforms.

How we built it

Frontend: React, Tailwind, Typescript
Backend: Selenium, GPT 4o, Python
Conversational Agent: LiveKit, OpenAI, Whisper (STT), Kokoro (TTS), Llama (Ollama) (LLM processing), Google Cloud

Our conversational agent orchestrates requests from users on the frontend and intelligent web navigation and processing in our backend (built with Selenium). Users can interact with both text and audio.

System Architecture

Challenges we ran into

Function Calling: Implementing and optimizing function calls to ensure smooth and accurate execution of user commands
Integration Complexity: Seamlessly integrating the backend server
Performance Optimization: Ensuring low and high responsiveness in voice interactions to maintain a fluid and efficient user experience

Accomplishments that we're proud of

Personalization Features: Developed strong personalization capabilities by hosting the application locally, allowing users to tailor the screen reader to their specific needs and preferences
Data Retrieval Accuracy: Achieved high accuracy in data retrieval and interpretation, ensuring reliable and precise information delivery to users
Extensive Platform Support: Successfully integrated TalkTuahWebsite with major browsers, operating systems, and popular applications, providing broad usability and accessibility

What we learned

Integrating Function and Local Hosting: Gained valuable insights into the complexities of integrating functional calls within a locally hosted environment, enhancing both performance and security
Flexibility Enhances Accessibility: Discovered that building a flexible and adaptable system is crucial for meeting diverse user needs and improving overall accessibility.

What's next for Untitled

Enhanced Multilingual Support: Expanding language options to cater to non-English speaking users and improve global accessibility
Advanced Contextual Understanding: Incorporating deeper contextual awareness to handle more complex user queries and provide more relevant assistance.
Integrating with Emerging Technologies: Exploring integrations with smart home devices and wearable technology to offer a more holistic accessibility solution.