APP ICON

Phoneme Gravity

Inspiration

Most modern speech-therapy and cognitive accessibility software suffers from a massive, systemic bias: it is designed exclusively around Western European phonologies (standard English or French).

Living in Morocco, I realized that children facing dyslexia or neurological speech-sound challenges across Africa are left with tools completely unsuited to their native tongue. Standard algorithms do not account for the rich pharyngeal fricatives of Afroasiatic languages (like Berber or Darija), the complex pre-nasalized double-consonant blends and implosives of Bantu systems (like Swahili or Zulu), or the fluid nasalized vocal glides of Niger-Congo dialects (like Wolof or Yoruba).

Phoneme Gravity was born to disrupt this. I wanted to build an immersive, real-time cognitive playground where speech sounds are modeled as celestial bodies bound by gravitational equations—specifically calibrated to the unique acoustic profiles of African linguistic families. By turning speech alignment into an interactive, real-time physics simulation, we transform frustrating speech correction into an intuitive, empowering space exploration.

How We Built It

The application is architected around a dual-engine architecture designed for high-frequency rendering and low-latency feedback loops:

The Neural Cognitive Parser: Powered server-side by the @google/genai SDK using Gemini Flash models to handle initial structural phonetic decomposition, grapheme-to-phoneme alignment, and multi-dimensional Levenshtein distance matrix calculations.
The Real-Time Physics Simulator: Built entirely in React 18, Vite, and styled with high-contrast Tailwind CSS, using motion for fluid visual rendering. Phonemes are treated as individual orbital nodes around a central "Gravity Core."
African Phonetic Gravity Calibrations:
- Bantu Epicycles: Employs synchronized double-orbit mathematical couplings to simulate the timing of pre-nasalized blends (e.g., /mb/ in Mombasa) and implosive offsets.
- Afroasiatic Jitter: Injects high-frequency micro-oscillations along the radial vectors to visualize pharyngeal friction and sibilant tension.
- Niger-Congo Flow: Maps fluid, expanding sine waves onto the orbital radii to replicate nasalized tonal glides and liquid omissions.

Mathematically, the position of each phoneme satellite node \( (x, y) \) at time \( t \) is governed by centripetal, wave-like, or jitter adjustments depending on the selected African Calibration:

\[ x(t) = \cos(\theta_i + \Delta\theta_{calibration}(t)) \cdot (r_i + \Delta r_{calibration}(t)) + x_{offset}(t) \] \[ y(t) = \sin(\theta_i + \Delta\theta_{calibration}(t)) \cdot (r_i + \Delta r_{calibration}(t)) + y_{offset}(t) \]

Challenges We Faced

Low-Latency Browser Audio Processing: Syncing browser speech capture with mathematical orbital updates required lightweight state management in React. We solved this by utilizing primitive state arrays and pre-calculating angular coordinates on every render frame.
Adapting AI Output for Physics: We had to strictly constrain the Gemini model to return precise, production-grade JSON strings containing both phonetic IPA representations and error indices, leaving zero room for markdown hallucinations that would otherwise break our physics engine's initialization loops.
Responsive Geometry: Designing an SVG-based orbital stage that scales fluidly across desktop screens and mobile viewports while keeping the coordinate space structurally consistent.

What We Learned

Acoustic Physics Representation: How to visually map acoustic parameters like vocal tension and friction to intuitive visual motion cues (like jitter, frequency, and wave amplitude).
The Power of Local Edge Inference: While our backup API uses Gemini, the system is designed to run completely offline with small models like Gemma-2B-Instruct using local inference, ensuring absolute privacy for children's voice data.
Linguistic Depth: Exploring the extreme acoustic richness of the continent’s languages, from the clicks of South Africa to the pharyngeal consonants of North Africa.