Inspiration
After exploring how technology has transformed assistive tools, we were drawn to AACs for nonverbal individuals. AACs give people a voice - yet many have remained largely unchanged for years, offering limited flexibility and personalization. We felt there was so much untapped potential to do better.
By bringing AI into AACs, we wanted to create something more responsive, expressive, and empowering - a system that adapts to the user, not the other way around. This mission resonated deeply with us, and we truly believed we could make a meaningful difference.
What it does
diaLEX is a next-generation AAC designed to help nonverbal users communicate faster, more naturally, and with confidence. It runs as an iPad-scale app familiar to existing AAC users: tap words or type a phrase, then press speak to be heard.
But diaLEX goes far beyond traditional AACs.
Using live camera input, diaLEX understands the user’s environment in real time. Objects in view instantly appear as clickable word cards, while AI-generated sentence suggestions adapt to what the user is seeing, doing, and experiencing. For users who rely on pointing to communicate, diaLEX recognizes pointed-at objects and brings them directly into the AAC—bridging physical intent and digital speech.
diaLEX also supports alternative expression through drawing. Users can sketch objects when spelling or vocabulary is difficult, and the app converts those drawings into meaningful word and image suggestions—meeting users where they are, not where technology expects them to be.
When conversations become challenging, diaLEX steps in. Its real-time question detector recognizes when a user is being asked something and offers an optional, dedicated question interface with relevant vocabulary, sentence starters, and AI-generated responses tailored to the moment. This helps users stay engaged in conversations without pressure or delay.
To ground everything in proven communication science, diaLEX includes a core vocabulary system of 100 expert-certified words that account for over 75% of daily communication, alongside customizable favorites and related word suggestions.
diaLEX doesn’t just give users words—it gives them context, confidence, and a voice in the moment it matters most.
How we built it
We built diaLEX as a cross-platform AAC system with a Flutter (Dart) frontend and an ASP.NET (C#) backend. A companion iPhone camera streaming service, developed in Swift, enables real-time visual context to flow directly into the AAC experience.
On-device Apple ML powers two core models: a YOLO-based object detection model and a hand recognition model for pointing and gesture awareness. Detected objects are extracted on the camera device and shared via peer-to-peer connectivity to ensure low-latency, privacy-preserving communication.
To enhance expressive power, we leverage OpenAI large language models to infer relevant words, sentence starters, and full responses based on environmental context, detected objects, and user intent. For users who prefer non-text input, we implemented a custom RNN that performs stroke-by-stroke analysis on drawings, converting sketches into meaningful word suggestions.
Throughout the app’s lifecycle, Flutter speech-to-text runs continuously in the background to detect incoming questions in real time. When a question is identified, diaLEX combines LLM reasoning, object analysis, and core vocabulary to generate interactive, personalized response options—allowing users to quickly and confidently participate in conversations.
Log in or sign up for Devpost to join the conversation.