The problem

About 240 million children live with a disability (UNICEF). Assistive tech reaches as little as 3% of those who need it in poorer countries (WHO/UNICEF). Apps that give a non-speaking child a voice cost hundreds of dollars, and a therapist has to build every board by hand.

What Aalap does

Take a photo. Get a board. Tap to speak. A photo of a meal table gives "I want water", "more", "too hot". A classroom gives "I need a break". Gemini Vision builds the board in the child's language and the phone speaks it aloud.

Beyond needs: feelings and dreams. 18 ready-made scenes cover daily life (morning, bedtime, medicine, shopping, travel, emergency) and the heart: "Mom, cuddle me", "I miss my dad", "I want to join in", "I can do it", "I have dreams too". Children are not only patients. They want to take part and grow up, and they need words for that.

One app, five combinable modes: Speech, Hearing (live captions in Chrome), Simple (fewer, bigger tiles for intellectual disability), Vision (tap once to hear, again to choose) and Switch (one-button scanning). A calm mode softens colours and removes animation.

Also: boards that learn the child's own words, sharing by QR code, export/import in the open Open Board Format (works with Cboard and CoughDrop), an SOS button, and 8 languages including right-to-left Urdu and Arabic. Ready-made boards work offline; building a board from a photo needs internet.

How I built it

A dependency-free JavaScript PWA with a small FastAPI backend that calls the Gemini API (vision with structured JSON output, and text-to-speech). About 1,300 bundled pictograms (Twemoji, CC-BY). Photos are processed in memory and never stored. Built solo during the hackathon (Nov 6-10, 2026).

Design choices

Core words (yes, no, help, more) stay in the same place on every board, so a child builds motor memory. This is standard AAC practice. Automated axe-core checks (WCAG 2.1 AA) report 0 violations, and the app is tested with 12 backend tests plus browser tests.

Challenges

  • Gemini model names were retired and renamed while I built this, so the server and client now try a chain of current models instead of one hard-coded name.
  • The best-known open pictogram set (ARASAAC) is non-commercial, so I used CC-BY Twemoji to keep the project clean to reuse.
  • Making a single UI work for very different needs (screen reader, switch scanning, simple mode, RTL) without it feeling cluttered.
  • Ordinary, unstaged photos: the prompt had to work for any photo (a plain desk, a dark room), so every board always includes everyday basics.

What I learned

Existing open-source AAC tools (Cboard, AsTeRICS Grid, CoughDrop) rely on boards made by hand. The new idea here is making the board for the moment the child is in, from one photo. I also learned that a communication tool has to carry feelings and hopes, not only requests.

What's next

Licensed pictogram sets, sign-language clips, native-speaker review of all translations (they are AI-assisted today), and pilots with special-education schools. A lightweight sync mode for low-bandwidth or satellite connections would let remote communities use it too.

Built With

Share this project:

Updates

Submission history