HearMe

interface
while using the app
while using the app
taking the permission to access the camera

💡 Inspiration

The inspiration for HearMe stemmed from observing the daily struggles the deaf community faces when interacting with essential businesses—such as retail stores, banks, and public transit. Currently, there is a massive business gap: companies lack the tools to effectively communicate with non-hearing customers, leading to poor customer experiences and lost economic opportunities. I realized that true independence for the deaf community means seamless, everyday transactions. HearMe was created to serve as an AI-powered life companion, empowering users to navigate daily activities independently while helping businesses bridge the communication barrier and provide truly inclusive services.

⚙️ What it does

HearMe goes beyond a typical sign-to-text application; it serves as a Contextual AI Interpreter tailored for everyday business and social interactions. By capturing quick sign-language shortcuts (tokens) through the camera, HearMe’s on-device model works seamlessly with a GenAI Agent to instantly translate them into natural, courteous everyday sentences in both Arabic and English. For businesses, this means their staff can understand and serve non-hearing customers effortlessly without needing to learn sign language, unlocking a more accessible and scalable customer experience.

🏗️ How I built it

Frontend & UI: Created entirely using Flutter, featuring a unique Neo-Agentic interface with glassmorphism styles, live token monitoring, and instantaneous visual responses.

Computer Vision (Edge AI): A tailor-made TensorFlow Lite (TFLite) model executed directly on the device. This guarantees extremely quick, low-latency sign recognition without transmitting heavy video frames to the cloud.

Reasoning Engine (GenAI): Raw tokens are selectively filtered and relayed to the Gemini 3.5 Flash API. Through sophisticated prompt engineering, Gemini functions as a contextual agent, comprehending the intent behind the shortcuts and providing natural conversational text suitable for public and commercial settings.

🚧 Challenges I ran into

Building a hybrid architecture that blends Edge AI with Cloud-based Generative AI as a solo developer presented significant hurdles. The primary challenge was minimizing latency; I had to ensure the TFLite model processed frames fast enough on-device before sending the token payload to the Gemini API, all while maintaining a smooth UI in Flutter. Additionally, crafting the perfect prompt engineering pipeline was tricky. I had to prevent the LLM from providing overly literal translations or hallucinating, ensuring it acted strictly as a contextual agent that outputs natural sentences suitable for real-world scenarios like shopping or commuting.

🏆 Accomplishments that I'm proud of

I am incredibly proud that my first-ever AI-powered project evolved from a conceptual idea into a fully functional, life-enhancing prototype. Bridging the gap between a local edge model (TFLite) and a state-of-the-art LLM (Gemini) independently—and wrapping it in a premium user interface—was a massive personal and technical milestone. Creating a scalable solution that directly addresses a real-world business gap while promoting inclusivity is my biggest win.

🧠 What I learned

I gained profound insights into the entire AI integration pipeline. I learned how to optimize Flutter apps for real-time computer vision, master prompt engineering to control GenAI outputs, and the absolute necessity of perseverance when independently debugging complex environment errors.

🚀 What's next for HearMe

Continuous Sign Recognition: Upgrading from static letters to dynamic, continuous sign language models.

Two-Way Integration: Adding Text-to-Speech (TTS) and Speech-to-Text (STT) for a seamless, bidirectional communication experience.

B2B Pilot Testing: Partnering with local retail stores or transit services to test HearMe in real-world business environments to validate its commercial impact.