Elio

Elio is a voice-first, multimodal messenger for blind and visually impaired users, powered by Gemini agents that understand context, use tools, and take real communication actions.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

Inspiration

More than 2 billion people worldwide live with some form of vision impairment. Modern messaging apps are increasingly built around visual interfaces, small controls, gestures, and complex navigation.

We asked ourselves a simple question:

What happens when a person who has always used modern technology loses the ability to see the interface?

They should not lose access to their conversations, friends, family, and digital independence.

That idea became Elio — an AI-first messenger designed around voice, accessibility, and natural interaction.

What it does

Elio is a fully functional modern messenger with an accessibility-first AI experience.

Instead of requiring users to visually navigate menus and buttons, a user can simply speak naturally:

  • “Elio, who messaged me?”
  • “Read my unread messages.”
  • “Open my chat with Alex.”
  • “Reply that I’m on my way.”
  • “Send it.”

Elio understands the request, determines the appropriate action, interacts with the messaging system, and provides spoken feedback to the user.

The assistant can be activated using voice or by pressing and holding anywhere on the interface.

Elio also includes a standard visual messenger experience, allowing blind, low-vision, and sighted users to communicate through the same product.

How we built it

Elio consists of a React Native iOS application connected to a NestJS backend running on Google Cloud.

At the center of the system is an AI agent powered by Gemini 3.5 Flash through Vertex AI and orchestrated using the Google Agent Development Kit (ADK).

When the user speaks, Elio captures raw audio and sends it through a real-time audio pipeline. Speech is converted into text, and the agent analyzes the user's intent, conversation state, contacts, pending actions, and messaging context.

The agent then produces structured actions such as:

  • opening a conversation;
  • reading messages;
  • resolving a contact;
  • preparing a reply;
  • requesting confirmation;
  • sending a message.

Actions are executed by the application rather than simulated by the AI.

For sensitive actions such as sending a message, Elio uses confirmation flows before execution.

We built the system around persistent WebSocket communication so the frontend can receive real-time agent events, task updates, message states, and assistant responses.

Our stack includes:

  • Gemini 3.5 Flash
  • Google Agent Development Kit (ADK)
  • Vertex AI
  • Google Cloud
  • Cloud Run
  • Google Cloud Speech-to-Text
  • Firebase Authentication
  • React Native
  • Expo
  • NestJS
  • PostgreSQL
  • WebSockets

Challenges we faced

One of the hardest challenges was building reliable voice interaction.

A voice-first messenger cannot require the user to repeat commands because the first part of their speech was clipped or because background noise was interpreted as speech.

We built a custom real-time audio pipeline with voice activity detection, audio pre-roll, silence detection, reliable WebSocket delivery, and persistent event sequencing.

Another major challenge was making AI actions safe and deterministic.

Elio does not allow the model to directly manipulate the application. Gemini produces structured decisions and proposed actions, which are validated before the application executes them.

We also had to rethink traditional messenger UX.

Instead of adding accessibility as a layer on top of a visual product, we designed the Accessible Mode around the assumption that the user may never look at the screen.

What we learned

Building Elio taught us that accessibility can fundamentally change how an AI product should be designed.

The most powerful use of an AI agent is not simply answering questions — it is giving a user the ability to operate software naturally.

We also learned the importance of separating AI reasoning from application execution. Combining Gemini's understanding with deterministic application actions allowed us to create an agent that is both flexible and controllable.

What's next

Our goal is to continue expanding Elio into a complete communication platform where users can manage messaging, calls, media, translation, location sharing, and other communication tasks naturally through voice.

We believe AI can make technology more accessible without creating a separate or simplified digital world for people with disabilities.

Elio is our attempt to make the same modern communication experience available to everyone.

What's next for Elio

Built With

Share this project:

Updates

Submission history