SonifyStem

Inspiration

STEM diagrams are everywhere in learning, but they are usually built for sight first. I wanted to make graphs, circuits, and biology diagrams easier for blind and low-vision students to explore independently, using sound instead of only text descriptions.

What it does

SonifyStem turns STEM diagrams into interactive spatial audio. As a user moves across a diagram, vertical position becomes pitch and horizontal position becomes stereo pan. Important labels ping and speak their name/description. Curves can also be played automatically so users can hear the shape of a graph, path, or circuit.

How we built it

I built SonifyStem with Next.js, React, TypeScript, Tailwind CSS, the Web Audio API, and the Web Speech API. The app has built-in sample diagrams, and uploaded diagrams can optionally be parsed with Gemini 2.5 Flash into structured JSON nodes and curves. We validate the model output and fall back to sample data if parsing fails.

Challenges we ran into

The hardest part was making the audio useful instead of overwhelming. I had to tune pitch, panning, collision detection, speech timing, and keyboard controls so the experience felt understandable. Another challenge was handling AI parsing safely, because uploaded diagrams can be messy or inconsistent.

Accomplishments that we're proud of

I'm proud that the app works offline with sample diagrams and does not break without an API key. I also built real accessibility features: keyboard navigation, speech announcements, high-contrast mode, live status updates, and curve playback using scheduled Web Audio ramps.

What we learned

I learned that accessibility is not just adding labels after the fact. The interaction itself has to be designed differently. I also learned a lot about sonification, spatial audio, and how structured AI output can connect visual diagrams to usable interfaces.

What's next for SonifyStem

Next, I want to improve diagram parsing accuracy, support more diagram types, add better customization for pitch/sound styles, and test with real blind and low-vision students so the tool becomes genuinely useful in classrooms.

Built With

• nextjs
• react
• tailwindcss
• typescript
• web-audio-api
• web-speech-api