ReadAR

• 

Inspiration

Reading shouldn’t be stressful, yet for millions of people with dyslexia or ADHD traditional text layouts create cognitive overload. Even for neurotypical readers, maintaining focus on dense text is harder than ever in an age of constant distraction.

We asked: What if reading could adapt to you, your focus, your attention, your pace instead of the other way around?

With Apple Vision Pro, we saw an opportunity to reimagine reading itself in spatial computing: to make text feel alive, responsive, and supportive.

That idea became ReadAR: an AI-powered reading companion that understands where you’re looking and what you need next.

What it does

ReadAR transforms static documents into immersive, adaptive reading environments.

Adaptive Focus Mode:

-Uses Vision Pro’s eye-tracking to detect where the user is looking.

-Dynamically highlights that line or paragraph, dimming surrounding text to reduce overload.

-Keeps the reader’s attention anchored — especially helpful for dyslexic or ADHD users.

Natural Voice Narration (ElevenLabs)

-Reads the current line or paragraph aloud in a natural, human-like voice.

-Activated with a simple pinch gesture or gaze dwell.

AI Comprehension Assistant (Gemini):

-When a user looks at a word or line and gestures, ReadAR can define, simplify, or summarize it.

-Gemini provides instant, context-aware explanations to enhance understanding.

Spatial Anchoring

-Text pages float in the user’s physical environment as spatial surfaces.

-Readers can pin a “definition wall,” a “summary board,” or a “focus page” around them for multi-panel comprehension.

Designed for Accessibility and Clarity

-Combines AI reasoning, spatial design, and adaptive visuals to reduce stress and enhance focus — “helping every mind read clearly, one line at a time.”

How we built it

Frontend (visionOS):

Built in SwiftUI + RealityKit to render text as spatial panels.

Integrated PDFKit for text parsing and region detection.

Used FocusEffect and HoverEffect APIs to detect eye gaze and highlight the focused line.

Implemented smooth blur and fade transitions to create a calming, adaptive reading environment.

Backend (AI):

Everything in swift

Gemini API (Google Generative AI) for contextual understanding, summarization, and simplification.

ElevenLabs API for lifelike text-to-speech narration.

Data Flow:

User opens a PDF → Vision Pro maps it as interactive text zones.

Gaze focus detected → highlight line.

Pinch gesture → triggers Gemini or ElevenLabs API.

AI response shown as floating text bubble or played aloud.

Challenges we ran into

The eye tracking not working since we can't collect data and therefore couldnt control the buttons with our eyes.

Accomplishments that we're proud of

Built one of the first AI-driven spatial reading experiences for Vision Pro.

-Seamlessly integrated navigation, natural voice narration and generative AI comprehension.

-Created a prototype that’s genuinely helpful for accessibility not just visually impressive.

-Achieved real-time reading assistance with minimal latency and a calming user experience.

-Designed a system that could easily expand into education, accessibility, and productivity tools.

What we learned

Spatial design changes cognition. Reading in 3D space, with adaptive focus, fundamentally changes how users engage with text.

Accessibility is innovation. Features built for dyslexia and ADHD improved focus for everyone.

AI + AR synergy is powerful. When AI reasoning meets intuitive spatial input, technology feels truly assistive, not intrusive.

visionOS development taught us to rethink traditional UI

What's next for ReadAR

Real-time eye-tracked highlighting with gaze dwell detection (for future VisionOS updates).

Offline reading mode with on-device summarization and speech synthesis.

Expanded Gemini modes — tone adjustment, emotional reading feedback, or AI reading coach.

Voice-controlled navigation for blind or low-vision users.

Classroom and workplace pilots to test comprehension improvement in real learning scenarios.

Integration with Apple’s accessibility APIs to make ReadAR a universal assistive reader.

Built With

• applevisionpro
• elevenlabs
• geminiapi
• swift
• swiftui
• visionos