EchoWorld

Hear your surroundings. EchoWorld uses Gemini AI to transform images into natural spoken descriptions, helping visually impaired individuals better understand their environment.

Comment

EchoWorld — Hear Your Surroundings

Inspiration

Imagine entering an unfamiliar room and not being able to see what is around you. Everyday tasks such as identifying furniture, locating doorways, or understanding the layout of a space can become challenging for individuals with visual impairments.

We wanted to explore how artificial intelligence could make environmental information more accessible.

That's why we created EchoWorld, an AI-powered accessibility tool that transforms visual information into natural, spoken descriptions.

Our goal was simple: Make the world easier to understand through sound.

What It Does

EchoWorld allows users to capture or upload an image of their surroundings and receive an AI-generated audio description of what is visible.

Rather than simply identifying objects, EchoWorld focuses on providing useful spatial context.

For example, when analyzing a room, EchoWorld might generate:

"There's a chair slightly to your left, a backpack directly ahead, and a doorway to your right."

The process consists of three main steps:

  1. Capture: The user takes a photo or uploads an image.
  2. Analyze: Google's Gemini AI identifies important objects and estimates their relative positions.
  3. Listen: ElevenLabs transforms the generated description into natural-sounding speech.

EchoWorld is designed to support environmental awareness and is not intended to replace mobility aids or established navigation techniques.

How We Built It

We developed EchoWorld as a web application, connecting computer vision, generative AI, and text-to-speech technology into a single workflow.

Technologies Used

  • Next.js: Full-stack web application and API endpoints.
  • React: Interactive, responsive user interface.
  • TypeScript: Type-safe frontend and backend development.
  • Google Gemini API: AI-powered image understanding and spatial scene analysis.
  • ElevenLabs API: Natural-sounding text-to-speech generation.
  • GitHub: Version control and team collaboration.
  • Vercel: Web application deployment.

Technical Architecture

Our application's core architecture follows this pipeline:

Image Capture / Upload → Gemini Vision → Structured Scene Analysis → Natural-Language Description → ElevenLabs Text-to-Speech → Audio Playback

We divided development into two primary responsibilities.

Frontend Development

Focused on building an intuitive interface that allows users to upload images, interact with the application, and listen to generated descriptions.

Backend and AI Integration

Focused on connecting Gemini's image-understanding capabilities with ElevenLabs' speech-generation API.

We designed two primary backend endpoints:

  • POST /api/analyze — Processes uploaded images and generates structured scene information and descriptions.
  • POST /api/speak — Converts generated descriptions into spoken audio using ElevenLabs.

This separation allowed us to work independently while maintaining a consistent integration between the frontend and backend.

Challenges We Faced

1. Building Within a Limited Timeframe

As a two-person hackathon team, we had limited development time and needed to prioritize the most impactful features.

We focused on delivering a functional core experience rather than introducing unnecessary complexity.

2. Turning Visual Information Into Useful Descriptions

Traditional image descriptions often identify objects without explaining where they are.

We wanted EchoWorld to go further by identifying approximate spatial relationships such as ahead-left, directly ahead, and ahead-right.

Designing prompts that encouraged Gemini to generate concise, relevant, and understandable scene descriptions was an important challenge.

3. Integrating Multiple AI Services

Connecting image analysis with speech synthesis required coordinating different API requests, response formats, and error-handling scenarios.

We designed a modular backend so each service could be tested and improved independently.

4. Balancing Accessibility With Accuracy

AI-generated image descriptions can sometimes be incomplete or inaccurate, especially when estimating distances or identifying obstacles.

We therefore focused on communicating approximate object positions instead of presenting EchoWorld as a precision navigation system.

Accomplishments We're Proud Of

  • Designing an accessibility-focused application with a clear real-world purpose.
  • Bringing together multimodal AI and natural-sounding speech synthesis.
  • Building a modular architecture that separates scene understanding from audio generation.
  • Creating a project that can be expanded beyond its initial hackathon prototype.
  • Collaborating as a two-person team under significant time constraints.

Most importantly, we explored how existing AI technologies can be combined to make visual information more accessible.

What We Learned

Building EchoWorld reinforced several important lessons.

AI is most useful when it solves a specific human problem. Rather than implementing AI simply for its novelty, we focused on using it to make environmental information more understandable.

We also gained practical experience integrating multiple AI APIs, designing structured responses, implementing full-stack endpoints, and coordinating frontend and backend development.

Finally, we learned that accessibility is not only about adding features. It requires carefully considering how users receive information, how much information they need, and how reliably that information can be communicated.

What's Next for EchoWorld?

Our hackathon prototype is only the beginning.

Future improvements could include:

  • Real-time camera analysis: Moving beyond individual photos toward continuous scene descriptions.
  • Improved spatial awareness: More precise estimates of object locations and distances.
  • Multilingual narration: Supporting users across different languages.
  • Voice commands: Allowing users to interact with EchoWorld without relying on visual controls.
  • Mobile optimization: Making EchoWorld more convenient for everyday use.
  • User testing: Working with visually impaired individuals to improve usability, accessibility, and reliability.

Our Vision

We believe artificial intelligence has tremendous potential to improve accessibility when applied thoughtfully.

EchoWorld represents our first step toward making visual environments more understandable through audio.

EchoWorld — Hear Your Surroundings.

Built With

Share this project:

Updates

Submission history