EduAudio Agent

Inspiration

Visually impaired and blind students face severe accessibility barriers when trying to navigate complex, unorganized, or poorly documented digital learning repositories. Traditional screen readers often read files linearly without context, making it extremely tedious for students to find specific lesson topics, verify file accuracy, or explore subject structures. We created EduAudio Agent to bridge this gap—using DataHub's metadata context graph to give blind students an accessible, structured, and voice-driven way to interact with educational data stacks.

What it does

EduAudio Agent is an accessible, context-aware AI platform that interacts with educational data repositories via DataHub's Model Context Protocol (MCP) and Agent Context Kit. Key capabilities include:

• Voice & Context-Aware Cataloging: Automatically scans raw audio lesson repositories, extracts topic hierarchies, and registers metadata schemas into DataHub so voice agents can fetch specific lessons instantly.
• Automated Accessibility & Quality Checks: Inspects incoming educational assets for missing transcripts, corrupted audio links, or broken schema tags, automatically flagging errors and writing updated descriptions back to the DataHub graph.
• Intelligent Audio Navigation: Lets blind students and educators query the dataset using plain English, accurately returning exact audio timestamps and lesson summaries without hallucinating.
• Hands-Free Voice Control: Built for WCAG AAA — an AI Teacher mode where students navigate pages by simply saying "next", "back", "repeat", or "stop", including voice barge-in to interrupt the teacher mid-sentence.

How we built it

• Metadata Context Engine: Integrated DataHub's open-source MCP Server to handle real-time schema reading, metadata lineage, and context updates.
• App & Backend: Built with Expo / React Native (web build via react-native-web) on Node.js, querying DataHub's GraphQL API and MCP tools.
• AI Agent Workflow: Configured an Agentic AI pipeline using open-source LLM integrations to process user queries, validate educational metadata against accessibility rules, and perform two-way sync with the DataHub catalog.
• Accessibility First: Web Speech API recognition with live transcript feedback, text-to-speech reading, push-to-talk, keyboard shortcuts (Space/M), a text command fallback, and WCAG AAA-compliant contrast.

Challenges we faced

• Two-Way Context Synchronization: Ensuring the agent doesn't just read DataHub metadata, but reliably writes back updated lesson tags, accessibility descriptions, and quality logs without schema drift.
• Latency Optimization: Keeping context retrieval fast enough so voice-driven queries feel responsive for visually impaired users.
• Voice Reliability: Taming Web Speech API quirks — Chrome's rate-limiting caused tight restart loops on network errors, so we added exponential backoff and a restart lock, and we made live voice commands fire instantly so navigation never waits for the speech session to end.

What we learned

• How DataHub's MCP Server acts as an essential context bridge, preventing AI agents from hallucinating when working across diverse data stacks.
• The critical importance of schema lineage and metadata governance when building accessible software for real-world communities.
• That accessibility must be engineered end-to-end: audio focus, echo control, fallback inputs, and interruptible speech are not optional extras.

What's next for EduAudio Agent

• Expanding DataHub custom connectors to index multi-lingual audio files (including Amharic educational content) for local school systems in Ethiopia.
• Native mobile voice triggers (push-to-talk, barge-in) on iOS/Android using on-device speech.
• Adding real PDF/document ingestion with automated transcript and schema verification.

Built With

• accessibility
• ai
• datahub
• expo.io
• express.js
• mcp
• node.js
• react-native
• speech-recognition
• typescript
• wcag-aaa