Project Title: Sahayak AI

Abstract:

All of us have had experiences with our grandparents / elderly relatives where they need help in trying to perform a simple task on their digital devices. A sizeable chunk of India's population, particularly the elderly and visually impaired, faces substantial barriers to digital inclusion. They struggle with the complexity / inaccessibility of modern graphical user interfaces (GUIs), creating a digital divide that limits independence and prevents access to essential online services like banking, e-governance, and e-commerce.

Sahayak AI is a voice-driven, agentic AI framework designed to dismantle these barriers. Our solution serves as an intuitive digital companion that allows users to operate any computer or application through natural voice-based conversation. By leveraging IBM's Agent Development Kit (ADK), we will build an intelligent agent capable of autonomously executing end-to-end tasks on GUI platforms.

At its core, Sahayak AI will be powered by an ensemble of IBM's Granite Vision models and other open-source models, utilizing their advanced natural language understanding and multimodal capabilities. This allows the agent to not only comprehend spoken commands in various Indian languages but also to visually perceive and interpret the screen's content in real-time.

Agentic Workflow in Action:

A user can state a complex goal like, "Book the next available train from Bangalore to Chennai for two adults." Sahayak AI autonomously orchestrates the entire workflow:

Deconstructs the user's intent into a sequence of actionable steps.
Executes these steps by directly interacting with the GUI – opening applications, navigating menus, and clicking buttons.
Perceives and understands the screen's content visually to make decisions in real-time.
Communicates its progress and results back to the user via voice for a transparent experience.

This project tries to utilize agentic AI to address a critical India-centric challenge. By translating user intent into direct GUI actions, we want to be able to empower digitally challenged users with digital autonomy, creating a more inclusive and accessible future.