Inspiration
The fundamental shortcut in computing is Ctrl + V: an immediate, friction-free transfer of information from one domain into another. Yet for more than 285 million blind and visually impaired (BVI) individuals worldwide, quantitative STEM information remains locked behind visual barriers.
When a screen reader encounters scientific charts, financial time series, or medical telemetry on platforms like arXiv, Bloomberg, or Wikipedia, it typically reads out high-level static alt-text (e.g., "Figure 2: Atmospheric carbon dioxide vs. global temperature anomaly"). This completely obscures rate of change, local extrema, volatility clusters, and critical inflection points.
We built SonifyV to establish sensory equality in quantitative research: an open-source, zero-friction acoustic interface where anyone can paste any scientific chart, time series, or dataset (Ctrl + V) and immediately explore it through musical frequency contour, binaural spatialization, speech telemetry, and physical tactile swell-paper matrices.
What it does
SonifyV transforms two-dimensional visual and tabular data into a rich, multi-sensory acoustic and tactile landscape:
Clipboard & Drag-and-Drop Ingestion (
Ctrl + V): Hooks directly into the system clipboard. Copy any chart screenshot or CSV table from arXiv, Bloomberg, or Wikipedia and pressCtrl + V(or drop the file onto the window) to parse coordinate paths directly into spatial audio channels.Psychoacoustic Sound Mapping with Mathematical Rigor:
- Y-Axis Value $\rightarrow$ Pitch: Values are normalized ($\bar{y} \in [0, 1]$) and mapped logarithmically across musical frequencies from $f_{\min} = 130.81\text{ Hz}\;(C_3)$ to $f_{\max} = 1046.50\text{ Hz}\;(C_6)$: $$f(y) = f_{\min} \cdot \left(\frac{f_{\max}}{f_{\min}}\right)^{\bar{y}}$$ Users can select between Pentatonic ($C, D, E, G, A$), Harmonic Minor, or continuous Microtonal pitch quantization to avoid auditory fatigue.
- X-Axis Timeline $\rightarrow$ Binaural Spatial Panning: Lateral stereo position is mapped continuously from left to right ear across time: $$p(x) = 2 \cdot \left(\frac{x - x_{\min}}{x_{\max} - x_{\min}}\right) - 1 \quad \text{where } p \in [-1.0, +1.0]$$
- Multi-Stem Acoustic Timbre: Dual-track comparative sonification allows tracking two correlated or diverging datasets simultaneously (e.g., Atmospheric $\text{CO}_2$ synthesized via 2-operator FM Bell synthesis vs. Temperature Anomaly synthesized via acoustic triangle waves) with independent Solo (
[S]) and Mute ([M]) controls.
In-Memory 16-bit PCM Audio Studio Exporter (
.wav): Utilizes an internalOfflineAudioContextto render synthesized spatial soundscapes into uncompressed 44.1 kHz 16-bit stereo.wavbinary files without needing external backend APIs: $$S_{16} = \max\left(-32768, \min\left(32767, \lfloor A \cdot 32767 \rceil\right)\right)$$Multilingual Speech Synthesis & Auto-Ducking: Integrated Web Speech API vocalizes exact coordinate coordinates, rate of change, and statistical summaries across 8 languages (English, Spanish, French, German, Hindi, Japanese, Mandarin, Portuguese) with automatic $-12\text{ dB}$ gain ducking during voice announcements.
Tactile Swell-Paper Matrix Export (
.svg): Exports high-contrast monochrome vector graphics calibrated with embossed dot-grid textures for thermal microcapsule paper (Swell-Touch/Capsule paper). When heated through a fuser, black toner areas swell into 3D tactile ridges, allowing users to feel the exact curve with their fingertips while listening to the audio sweep.Zero-Mouse WCAG 2.1 AAA Accessibility: Full keyboard control:
[Space]for sweep playback,[←/→]for single-step coordinate scrubbing,[D]for datum coordinate whispers,[M]for instant peak-extrema jump, and[B]for high-contrast accessibility HUD.
How we built it
- DSP Audio Synthesis Engine: Built from scratch using the native Web Audio API, featuring a 2-operator FM synthesizer, dynamic ADSR envelopes, low-pass biquad filters, and stereo panning nodes delivering sub-5ms latency.
- WAV Binary Encoder: Programmed direct little-endian byte array packing (
RIFF,WAVE,fmt,datachunks) to export raw 16-bit stereo PCM audio directly in the client browser. - Spectral Analysis Oscilloscope: Designed a real-time 60 FPS HTML5 Canvas rendering Fast Fourier Transform (FFT) frequency spectrum and audio wave telemetry.
- Interface & Typography: Engineered in React 18, TypeScript, and Tailwind CSS with a distraction-free slate/zinc precision design system.
- Automated Verification: Authored 14 unit tests in Vitest verifying monotonic frequency quantization, CSV parsing, multi-stem audio balance, and localization mappings.
Challenges we ran into
- Auditory Fatigue vs. Mathematical Precision: Continuous unquantized frequency sweeps quickly cause ear fatigue and disorientation. However, coarse quantization can introduce step artifacts that obscure subtle inflections. We resolved this by formulating discrete pentatonic and harmonic scale tables that preserve mathematical monotonicity while delivering musical contour.
- Real-Time Speech & Frequency Intermodulation: When verbal coordinate telemetry played concurrently with rapid pitch sweeps, acoustic masking caused speech comprehension to drop. We engineered an auto-ducking gain automation curve that dynamically attenuates the audio oscillators by $-12\text{ dB}$ whenever the speech synthesizer speaks.
- In-Browser Audio Compilation Without Server Latency:
Generating uncompressed studio
.wavfiles on the client required coordinating asynchronous audio graph rendering with direct typed-array binary serialization, ensuring zero server latency and total user data privacy.
Accomplishments that we're proud of
- Zero-Latency In-Browser Architecture: Completely client-side; no telemetry leaves the user's browser, ensuring privacy and offline capability.
- Full Sensory Dual-Modality: Seamlessly bridging real-time stereo spatial audio with physical tactile swell-paper vector printing.
- Rigorous WCAG 2.1 AAA Navigation: True zero-mouse operation designed from the ground up for assistive technology.
- Production Deployment & Green CI: Fully automated testing with 14 passing unit tests deployed on GitHub Pages.
What we learned
- Human Pitch Perception is Logarithmic, Not Linear: Weber-Fechner psychoacoustic laws demonstrate that human frequency discernment follows proportional ratios ($\Delta f / f = \text{constant}$). Mapping numerical values linearly to Hertz produces an auditory illusion where lower data values sound compressed and higher values sound disproportionately exaggerated. Logarithmic scaling is mathematically essential for truthful non-visual data comprehension: $$\log_2\left(\frac{f_2}{f_1}\right) = \text{Musical Interval in Octaves}$$
- True Accessibility Accelerates Usability for Everyone: Designing an interface that can be operated purely via keystrokes and audio feedback created an extraordinarily efficient, distraction-free environment that sighted researchers can use for peripheral eyes-free monitoring.
- Multimodal Sensory Redundancy: Pairing spatialized auditory sweeps with tactile relief graphics provides cognitive parity with visual charts. Sound excels at communicating overall velocity, trends, and sudden anomalies, while tactile graphics provide static spatial reference.
What's next for SonifyV:Auditory Data Interface for Non-Visual STEM Analysis
- Browser Extension: A lightweight extension that activates upon pressing
Ctrl + Vover any online scientific paper (e.g. arXiv, Nature) to sonify embedded figures inline. - 3D Volumetric Sonification: Expanding from 2D curves to 3D scatter plots and tensor fields using Web Audio HRTF (Head-Related Transfer Function) spatial audio.
- Direct Refreshable Braille Integration: Streaming coordinate telemetry directly to physical refreshable Braille displays in real time.
Built With
- html5
- offline-audio-context
- react
- tailwindcss
- typescript
- vite
- vitest
- web-audio-api
- web-speech-api