HearMe

interface
while using the app
while using the app
taking the permission to access the camera

💡 Inspiration

The inspiration for HearMe stemmed from a tragic and entirely avoidable incident in real life. At a university graduation event, a deaf student suddenly collapsed and was taken to the hospital. Unable to verbally communicate, he frantically used sign language to convey his medical condition to the doctors. Unfortunately, none of the medical staff could understand him. Because the medical team couldn't obtain his medical history or comprehend his urgent symptoms, he sadly lost his life. By the time his family arrived and clarified that he was deaf, it was already too late.

This devastating event left me with a crucial question: What if there had been just one person in that room who understood sign language? That question fueled my purpose. I created HearMe to make sure that no one ever loses their voice or their life due to a communication barrier.

⚙️What it does

HearMe goes beyond a typical sign-to-text application; it serves as a Contextual AI Interpreter. It connects the deaf community with the hearing world in real-time. Rather than providing strict, literal translations, HearMe captures quick sign-language shortcuts (tokens) through the camera and employs a GenAI Agent to promptly transform them into full, natural, and courteous everyday sentences in both Arabic and English.

🏗️ How we built it

Frontend & UI: ** Created entirely using Flutter, featuring a unique Neo-Agentic interface with glassmorphism styles, live token monitoring, and instantaneous visual responses. *Computer Vision (Edge AI): * A tailor-made TensorFlow Lite (TFLite) model executed directly on the device. This guarantees extremely quick, low-latency sign recognition without transmitting substantial video frames to the cloud. **Reasoning Engine (GenAI): This is where the real innovation occurs. Raw tokens are selectively filtered and relayed to the Gemini 3.5 Flash API. Through sophisticated prompt engineering, Gemini functions as a contextual agent, comprehending the meaning behind the shortcuts and providing natural conversational responses.

🚧 Challenges we ran into

We engineered a hybrid architecture blending Edge AI with Cloud-based Generative AI:

** Frontend & UI:** Developed purely in Flutter, featuring a custom Neo-Agentic UI with glass morphism effects, live token tracking, and real-time visual feedback. ** Computer Vision (Edge AI):** A custom TensorFlow Lite (TFLite) model deployed directly on the device. This ensures ultra-fast, low-latency sign detection without sending heavy video frames to the cloud. ** Reasoning Engine (GenAI): ** This is where the magic happens. Raw tokens are dynamically filtered and sent to the Gemini 3.5 Flash API. Using advanced prompt engineering, Gemini acts as a contextual agent, understanding the intent behind the shortcuts and returning natural conversational text.

🏆 Accomplishments that we're proud of

I am incredibly proud that my first-ever AI-powered project evolved from a conceptual idea into a fully functional, life-saving prototype. Bridging the gap between a local edge model (TFLite) and a state-of-the-art LLM (Gemini) independently—and wrapping it in a premium user interface—was a massive personal and technical milestone.

🧠 What we learned

I gained profound insights into the AI integration pipeline. I learned how to optimize Flutter apps for real-time computer vision, master prompt engineering to control GenAI outputs, and the absolute necessity of perseverance when facing complex environment errors.