A11ySTEM — Turning STEM Diagrams into Spatial Audio

Inspiration

Over 90% of secondary and university STEM curricula rely heavily on visual graphics: circuit schematics, metabolic reaction loops, differential calculus curves, and data structures.

For sighted students, a diagram communicates complex relationships at a glance. However, for millions of blind and low-vision students worldwide, these visual materials create significant barriers to learning.

Standard screen readers such as NVDA, JAWS, and VoiceOver often treat SVGs as flat strings of XML tags or reduce them to unhelpful descriptions like "Figure 3: Circuit diagram."

Commercial tactile graphics embossers cost $3,000–$10,000+, require specialized microcapsule paper, and can take weeks to produce materials. This makes them impractical for fast-paced homework and remote learning.

We asked:

What if blind students could perceive the two-dimensional spatial topology of any STEM diagram directly through sound?

This question inspired A11ySTEM, a zero-install, browser-native platform that translates visual schematics into binaural spatial audio, harmonic pitch frequencies, and interactive 2D keyboard navigation.

What It Does

A11ySTEM transforms visual diagrams into interactive, audible mental models through five core features.

1. Binaural Spatial Audio Mapping

We translate diagram coordinates and component properties into distinct audio signals.

• Horizontal (X-Axis): Maps spatial position to stereo panning, ranging from full left (-1.0), through center, to full right (+1.0).
• Vertical (Y-Axis): Maps vertical position to a logarithmic harmonic pitch scale, ranging from C3 (130.81 Hz) to C6 (1046.50 Hz).
• Component Timbre: Different waveform shapes represent different component types:
  • Pure sine waves for resistors and metabolites.
  • Warm triangle waves for sources and data points.
  • Capacitive ramps for capacitive components.
  • Deep 65 Hz sub-bass for ground references.

Smooth exponential attack and decay envelopes prevent audio clicks and pops.

2. 2D Cardinal Directional Navigation

Instead of forcing users to read diagrams in a linear sequence, A11ySTEM enables spatial navigation using standard keyboard arrow keys:

• Left and Right
• Up and Down

An angular distance algorithm identifies the nearest neighboring node in each cardinal direction.

The navigation score is calculated as:

Score = Δx + 1.8 × Δy

This allows users to explore diagrams according to their actual spatial arrangement.

3. Continuous Audio Sweep Scanner

Pressing the Spacebar launches a smooth, three-second left-to-right audio sweep across the entire canvas.

This provides an overview of the diagram's spatial topology before users explore individual components.

4. Multi-Disciplinary STEM Suite

A11ySTEM includes four preloaded university-level STEM examples:

Analog Electronics — Inverting Operational Amplifier

An interactive circuit featuring an operational amplifier, virtual ground, and feedback loop.

Av = -Rf / Rin = -10

Differential Calculus — Damped Harmonic Oscillation

Explores the function:

f(t) = e^(-0.15t) × cos(2t)

Includes tangent slope sonification and zero-crossing detection.

Cellular Biochemistry — Citric Acid (Krebs) Cycle

An eight-step metabolic loop representing the interconnected stages of cellular energy production, with enzyme-substrate energetics.

Computer Science — Balanced Binary Search Tree (AVL)

Explores tree structures through spatial audio. In-order traversal produces an ascending pentatonic musical scale.

5. Instant SVG Ingestion and ARIA Export

Educators and students can paste SVG schematics or upload vector files.

The parser extracts component centroids, calculates 2D spatial relationships, and generates enhanced SVGs with embedded accessibility metadata, including:

• role="treeitem"
• aria-label
• <desc> descriptions

This makes existing vector diagrams more accessible without requiring educators to recreate them from scratch.

How We Built It

Frontend

• React 19
• TypeScript
• Tailwind CSS v4

The interface uses a high-contrast dark obsidian (#080C14) and luminous amber (#F59E0B) palette, designed around WCAG 2.1 AAA contrast requirements.

Acoustic Synthesis Engine

Built entirely using the native Web Audio API, without prerecorded audio samples.

Core components include:

• AudioContext
• StereoPannerNode
• GainNode
• OscillatorNode

All audio synthesis runs directly in the browser, enabling offline operation with zero network latency.

Speech and Screen Reader Integration

We integrated the Web Speech API with automatic speech queue cancellation to prevent announcement delays during rapid keyboard navigation.

An aria-live="polite" region provides real-time accessibility telemetry.

Topological Graph Engine

A custom TypeScript geometry engine processes SVG coordinate attributes to construct directed adjacency matrices and cardinal-direction neighbor projections.

Testing and Deployment

• Vitest for unit testing.
• Playwright for end-to-end browser automation.
• Tested on desktop (1440px) and mobile (390px) viewports.
• Deployed globally through Vercel.
• Source code maintained on GitHub.

Challenges We Ran Into

1. Preventing Auditory Cognitive Overload

Our early prototypes attempted to play every connected wire simultaneously. The result was an overwhelming and chaotic audio experience.

We addressed this by separating the experience into two layers:

• Macro: Audio Sweep — Provides an overview of the diagram.
• Micro: Focal Reticle — Plays a single active tone with directional audio hints during keyboard navigation.

2. Audio Popping and Phase Clicks

Rapid oscillator starts can produce harsh audio pops when gain changes instantaneously.

We implemented a 15ms exponential gain attack and a 200ms decay curve to produce smoother audio transitions.

3. SVG Group Flattening

Many CAD and drawing applications, including Inkscape and Illustrator, organize visual elements inside deeply nested <g> containers.

We developed a recursive selector that extracts individual visual shapes while inheriting parent group IDs as subsystem metadata.

Accomplishments We're Proud Of

100% Benchmark Accuracy

We validated A11ySTEM against 15 external benchmarks, including:

• Texas Instruments NE555 official datasheets.
• W3C SVG 1.1 electrical standards.
• NIH PubChem chemical structure graphs.

Our evaluation achieved 100% topological accuracy with an average processing latency of 0.8ms.

Dual-Experience Ergonomics

A11ySTEM supports collaboration between sighted educators and blind students.

• Sighted educators can observe animated sonar pulse rings, directional connection vectors, and real-time panning gauges.
• Blind students can navigate diagrams using audio cues and standard keyboard shortcuts.

Both users interact with the same underlying diagram.

Zero Hardware Barrier

A11ySTEM provides an alternative to expensive specialized tactile graphics equipment.

It runs in a standard web browser using ordinary computer speakers and keyboard input, without requiring dedicated hardware.

What We Learned

Accessibility Is an Architectural Decision

Designing for blind users from the beginning forced us to rethink how diagrams should be represented and explored.

We prioritized predictable navigation, reliable state management, and immediate feedback instead of treating accessibility as an additional interface layer.

Psychoacoustics as an Information Channel

Human hearing can distinguish subtle differences in pitch, stereo positioning, and timbre.

Through sonification, sound can communicate structural density, spatial position, and physical relationships without requiring users to rely entirely on lengthy text-to-speech descriptions.

What's Next for A11ySTEM

AI-Powered Photo-to-Schematic OCR

Integrating multimodal vision models to recognize diagrams from photographs of classroom whiteboards and convert them into interactive audio topologies.

WebHID Refreshable Braille Display Integration

Adding support for refreshable Braille displays through WebHID, allowing students to explore pin-level information through both touch and sound.

Expanded Scientific Domains

Expanding beyond 2D STEM diagrams into:

• 3D protein structures and PDB files.
• Complex planetary orbital mechanics.
• Additional scientific and engineering visualization domains.

Our long-term goal is to make visual STEM knowledge accessible through multiple sensory channels, directly within the browser.

Built With

• accessibility
• react
• svg
• tailwindcss
• typescript
• vite
• web-audio-api
• web-speech-api