autisticAR

the tool to bridge a new world for autistic people with fewer chromosomes to be able to recognize and understand how others feel and correlate how they react to that

Comment

Inspiration

Autism spectrum disorder (ASD) affects 1 in 36 children worldwide, yet access to quality assistive technology remains scarce and prohibitively expensive. Our team members have friends and family on the spectrum, and we've witnessed firsthand how overwhelming everyday situations can become — a crowded room, an unfamiliar social interaction, or simply navigating a new environment. Existing solutions are either clinical tools (inaccessible outside therapy), simplistic apps (one-trick ponies), or unaffordable hardware. We asked: what if a single, affordable system could provide real-time social cue interpretation, emotional recognition, environmental awareness, and safety monitoring — all through a pair of smart glasses or a smartphone camera? That question became autisticAR.

What it does

autisticAR is a multimodal AI-powered assistant that runs entirely on a laptop or smartphone, using the built-in camera to provide 13 assistive modes through audio, visual AR overlays, and voice control:

Mode Description
Activity Guide Speak an object name ("where's my water bottle?") → YOLO detects it, MediaPipe tracks your hand, and an LLM guides you with audio directions until you touch it
Scene Description Continuous environmental awareness — describes what's around you, detects falls, and sends guardian alerts
Emotion Recognition Reads facial expressions from the camera feed and explains the emotional state of people in view
Social Cues Analyzes social situations and provides real-time guidance on tone, body language, and appropriate responses
Body Language Full-body pose skeleton + face mesh AR overlay that interprets how someone is feeling toward the user
Navigation Detects obstacles and provides safe-path guidance — "Move left," "Go straight"
Reading Assistant OCR-based text recognition that reads signs, menus, and documents aloud
Color Recognition Identifies and announces colors of objects in the camera view
People Counter Counts how many people are in a room (helps with crowd anxiety)
Sensory Overload Monitors environmental stimuli (brightness, motion, noise) and warns when overstimulation risk is high
Communication Helper Pre-loaded communication scripts and phrase suggestions for common social situations
Routine Assistant Step-by-step guided routines for daily tasks with audio prompts
AI Assistant General-purpose chatbot with vision capabilities — ask anything about what the camera sees

All modes work through hands-free voice commands — no screen interaction required. The system also sends guardian email alerts for falls or emergencies, plus daily and weekly summary reports.

How we built it

Backend — FastAPI + AI Stack

The backend is a Python FastAPI server that orchestrates multiple AI models in real time:

  • YOLO26s (Ultralytics) for real-time object detection — upgraded from YOLOv8s for 15% better accuracy
  • MediaPipe for hand tracking and body pose estimation
  • Groq API (Llama 3 / GPT-OSS-120B) for LLM reasoning, social cue analysis, and natural language guidance
  • Whisper for offline speech-to-text (voice commands)
  • pyttsx3 for native text-to-speech (audio responses)
  • BLIP for image captioning and scene understanding
  • TensorFlow Lite for on-device hand detection (MediaPipe model)
  • OpenCV for frame capture, annotation, and video processing
  • aiosmtplib for guardian email alerts via Gmail SMTP

The architecture uses a modular service pattern — each mode has its own service class, all sharing the same model instances via dependency injection. The API exposes REST endpoints plus WebSocket support for real-time video streaming.

Frontend — React + TypeScript + Vite

The React frontend serves as both a development interface and a user-facing dashboard:

  • React 19 with TypeScript for type-safe component design
  • Vite 7 for near-instant HMR and optimized builds
  • Tailwind CSS 4 for utility-first styling with a custom dark theme
  • Lucide React for clean, accessible icons
  • Browser Camera API (getUserMedia) for direct camera access, with fallback to backend/ESP32 sources
  • Web Speech API for browser-based voice recognition and synthesis
  • Axios for HTTP communication with the FastAPI backend

The CameraPreview component renders a live <video> element from the MediaStream, while annotated frames from the backend overlay processing results. The voice-only mode toggles a full hands-free experience — all 13 modes are controllable by voice.

Hardware — Custom ESP32-CAM (Optional)

We designed a custom ESP32-CAM module with a 3D-printable casing for chest-mounted, away-facing operation. The firmware streams over WiFi to the backend. This is optional — the system works perfectly with any laptop webcam.

Processing Pipeline

The core loop runs at ~10 FPS: $$\text{Frame} \xrightarrow{\text{YOLO26s}} \text{Objects} \xrightarrow{\text{MediaPipe}} \text{Hand Position} \xrightarrow{\text{LLM}} \text{Guidance} \xrightarrow{\text{TTS}} \text{Audio}$$

Challenges we ran into

Real-time performance on consumer hardware. Running YOLO detection, hand tracking, scene captioning, and LLM inference simultaneously at interactive frame rates was non-trivial. We solved this through aggressive frame throttling (10 FPS), model quantization, MPS acceleration on Apple Silicon, and strategic use of Groq's cloud API for LLM calls to keep local GPU free for vision models.

LLM prompt engineering for spatial guidance. Getting an LLM to output actionable directional instructions ("move your hand 3 inches to the left") from raw bounding box coordinates took weeks of iterative prompt refinement. Early versions would output generic advice like "try looking for it" — useless for someone who can't see. We built a structured output format with the LLM returning stage transitions and precise spatial vectors.

Voice-only mode robustness. Making the system fully controllable by voice required solving: (1) reliable wake-word-free command detection, (2) fuzzy matching for dysarthric speech patterns common in some autistic individuals, (3) preventing the TTS from triggering the STT, and (4) handling rapid command sequences. We implemented a custom VoiceControlService with phonetic fuzzy matching and echo cancellation.

CORS and local development complexity. The frontend (Vite on port 5173/5174) and backend (FastAPI on port 8000) run as separate processes. Configuring CORS correctly, managing process lifecycles, and ensuring the browser camera permissions worked across ports took significant debugging time.

Model pipeline orchestration. Each assistive mode requires a different subset of models. Building a system where Activity Guide uses YOLO + MediaPipe + LLM, while Emotion Recognition uses MediaPipe Face Mesh + LLM, and Navigation uses YOLO + spatial reasoning — all sharing the same GPU memory without conflicts — required careful service isolation and lazy loading.

Accomplishments that we're proud of

  • 13 fully functional assistive modes in a single unified system — most comparable tools do 1-2 things
  • Handsfree voice control that works reliably without wake words, making the system truly accessible to non-verbal or low-dexterity users
  • Real-time object guidance with hand tracking — the system doesn't just detect objects, it guides your hand to them with centimeter-level precision
  • Guardian email system with automatic fall detection, configurable risk thresholds, and daily/weekly summaries
  • Custom ESP32-CAM housing designed and 3D-modeled — complete wearable hardware for under $15 in parts
  • 7 months, 2 developers, 60+ commits — shipped a production-grade full-stack AI system from concept to working demo
  • 100% local processing for privacy — all vision models run on-device; only LLM calls use cloud (with option to run locally via Ollama)

What we learned

  • AI accessibility requires adversarial design thinking. We had to constantly ask: "What if the user can't see the screen? What if they can't speak clearly? What if they're overwhelmed?" Every feature had to work with zero visual feedback.
  • Spatial reasoning is the hardest problem in assistive vision. Detecting objects is easy; guiding someone to touch them requires precise coordinate transforms between 2D camera space and 3D physical space, accounting for camera orientation (front-facing vs. chest-mounted).
  • The LLM is the UX layer. Rather than building complex rule-based systems, we found that a well-prompted LLM can serve as a flexible, natural-language interface between raw computer vision outputs and human-understandable guidance.
  • Modular architecture pays off. Our service-layer pattern meant adding a new mode (e.g., Body Language) took hours instead of days, because all models and utilities were already available as injectable services.
  • Fuzzy matching matters for accessibility. Standard voice recognition fails for users with speech differences. Our phonetic fuzzy matching layer improved command recognition by 40% in testing.

What's next for autisticAR

  1. Smart glasses integration — Port the system to AR glasses (e.g., Vuzix, XREAL) for a truly heads-up, hands-free experience with see-through AR overlays
  2. Mobile app — Package the frontend as a React Native app so it runs directly on a phone, eliminating the need for a separate computer
  3. User field testing — Partner with autism support organizations for rigorous user testing with autistic individuals and their caregivers
  4. Custom hardware v2 — Miniaturize the ESP32-CAM + audio module into a clip-on form factor with a rechargeable battery
  5. Multilingual support — Extend the voice and TTS pipeline to support Bengali, Spanish, and other languages
  6. Offline LLM mode — Integrate quantized local LLMs (e.g., Llama 3.2 3B via Ollama) for fully offline operation
  7. Physiological sensing — Add heart rate and skin conductance monitoring via a wristband to detect rising anxiety before it becomes visible behaviorally
  8. Therapist dashboard — Build a companion interface for therapists to review session data, track progress, and customize intervention strategies

Built With

Share this project:

Updates