VoxSurf.ai

Inspiration

We noticed that many visually-impaired or blind users struggle to browse the web in a fluid, natural way: web pages are built visually and keyboard or screen-reader navigation is often cumbersome. We asked: what if the browser itself could listen, speak, and assist the user as they surf the web — bringing voice + AI agents into the browsing experience? That’s how VoxSurf.ai was born: to empower blind users to “hear” the web, navigate by voice, and truly interact with web content rather than just passively consume it.

What it does

VoxSurf.ai is a voice-powered, agentic web accessibility platform that allows users to surf the internet through speech. By combining automatic speech recognition, real-time LLM reasoning, and browser automation, users can perform actions like navigating websites, summarizing pages, filling forms, or generating insights without typing. It acts like an autonomous browsing companion powered by multimodal intelligence.

How we built it

The architecture combines several powerful components:

Deepgram: For streaming speech recognition (ASR) to convert voice commands into accurate text transcriptions.

Groq and Gemini: For real-time response generation and contextual reasoning, including intent parsing and task execution.

Fetch: For web data retrieval and dynamic page content analysis.

Python backend: The backend (backend/src/llms and backend/src/agents) manages AI model orchestration, intent classification, prompt chaining, and voice command routing.

Frontend (root level): HTML and JS components for the user interface with a minimal footprint, providing real-time voice capture and interaction feedback. The entire application runs locally as a desktop app packaged with PyInstaller, ensuring privacy-focused offline execution.

Challenges we ran into

Designing low-latency pipelines for streaming voice input was complex — balancing speed and model inference response across Groq, Gemini, and Deepgram was a challenge. Maintaining consistent performance across different network conditions and browsers required careful optimization and caching. Integrating multiple AI APIs, managing tokens, and synchronizing context between speech and web state also presented engineering hurdles.

Accomplishments that we're proud of

Built a seamless voice-controlled browsing system combining ASR, LLMs, and autonomous web agents.

Achieved near real-time, natural-feeling voice-to-action latency.

Enabled offline execution through a desktop packaging workflow.

Developed modular backend agents for autonomous task reasoning and execution, allowing rapid extensibility for future integrations.

What we learned

Building multimodal AI systems requires carefully managing data flow between voice, vision, and reasoning streams. Latency optimization and human-in-the-loop design vastly improve user experience. We also learned how critical prompt engineering and chaining are for natural agentic task execution — especially when combining different APIs with varying response structures.

What's next for VoxSurf.ai

The next steps include integrating vision-capable agents (for webpage screenshot understanding), memory modules for persistent personalization, and multi-agent collaboration, where one agent narrates web content while another interacts contextually. We also plan to expand support for more languages, browser extensions, and mobile accessibility apps, furthering VoxSurf.ai’s mission to make digital experiences universally accessible.