AyuSetuXR

Inspiration

We live in an age of hyperconnectivity — but our mental connection to ourselves is fading. Screens dominate our attention spans, and the cognitive fatigue that follows is invisible yet constant. At HackHarvard, we asked: Can technology restore balance instead of breaking it? That question led to AyuSetuXR — “Ayu” meaning life and “Setu” meaning bridge. It’s an extended reality (XR) app that bridges the human mind and digital awareness through real-time AI feedback — helping people monitor focus, fatigue, and stress in a natural, immersive way.

What it does

AyuSetuXR uses your device camera to detect real-time facial microexpressions and eye behavior, interpreting them into cognitive metrics like: Attention Score (focus stability) Fatigue Level (eye strain, blink rate) Emotional balance (gaze and posture cues) These metrics are analyzed through Gemini AI hosted on Cloudflare Workers, which transforms raw data into meaningful insights. Then, ElevenLabs gives those insights a human voice, delivering real-time multilingual feedback like: “You seem slightly tired. Take a short break.” “Your focus is great — keep it up.” In short, it’s an AI-powered mirror for your mind, guiding you toward awareness and balance within XR environments.

How I built it

Frontend (XR Layer): Built with React Native + Expo for cross-platform XR experiences. Face Tracking: Implemented via MediaPipe FaceMesh to detect 3D facial landmarks. AI Reasoning: Metrics are sent to a Cloudflare Worker, which uses Gemini Pro for cognitive interpretation. Voice Output: Integrated ElevenLabs Multilingual API to convert AI insights into speech in the user’s language. Visualization: Overlays in real time — metrics like focus %, blink detection, and gaze direction animate directly on screen. The entire system runs lightweight, real-time analytics — designed to work even on mobile devices using edge-optimized architecture.

Challenges I ran into

Getting MediaPipe to work in React Native with live camera frames was complex, especially across devices. Synchronizing AI reasoning (Gemini) and audio synthesis (ElevenLabs) via Cloudflare Workers required async optimization. Managing real-time frame processing without TensorFlow (for EAS build stability) meant re-engineering performance. Debugging camera permission crashes and Expo build configurations under time pressure was tough.

Accomplishments that I'm proud of

Built a working XR interface that connects real-world expressions to cloud-based AI reasoning in real time. Created a multilingual adaptive feedback system — a voice that understands and speaks in the user’s native tone. Developed a lightweight, ethical framework for monitoring cognition without storing or sharing personal data. Got the system running cross-platform and deployed through Expo + EAS + Cloudflare pipeline.

What I learned

How to integrate AI reasoning, real-time computer vision, and voice synthesis in a seamless experience. The power of edge computing (Cloudflare Workers) for low-latency, privacy-respecting AI applications. Importance of designing AI that feels human — not analytical, by focusing on emotional UX. And perhaps most importantly: building with purpose makes even complex technology feel meaningful.

What's next for AyuSetuXR

EEG + Wearable Integration: Connect brainwave or heart-rate sensors for deeper physiological awareness. AR/XR Mode Expansion: Port to Vision Pro, Meta Quest, and WebXR for full immersion. Mental Health Extension: Offer guided focus and mindfulness sessions based on live attention data. Ultimately, AyuSetuXR is not just an app — it’s a human augmentation layer for the next generation of conscious technology.

Built With

• cloudflare
• elevenlabs
• gemini
• react-native
• supa
• typescript