Phoneme Orbit

Phoneme Orbit: Next-gen offline DevSecOps auditing, powered by local Edge ML constraints and sub-millisecond phonetic DSP resonance.

Comment

Inspiration

As a 13-year-old software and AI developer aiming for the Apple WWDC 2027 Swift Student Challenge, my core mission has been building Phoneme Orbit: a privacy-focused education and cognitive accessibility platform.

We built Phoneme Orbit as a unified workspace to address this. It pairs a low-latency DSP (Digital Signal Processing) syllable synthesizer—helping dyslexic and speech-impaired learners practice phoneme-grapheme correspondences offline—with an Elite SAST and Secrets Scanner backed by GitLab Duo AI to audit and secure local codebase architectures.

What it does

Phoneme Orbit operates as an advanced, dual-purpose cockpit:

  1. The Syllable Synthesizer (GitLab Orbit Duo): Generates human voice formants completely client-side in real-time. By utilizing the Web Audio API, users can manually dial in the fundamental pitch ($F_0$), throat opening resonance ($F_1$), and tongue placement ($F_2$).
  2. Elite SAST Router & Security Radar: Developers inputs a GitHub or GitLab repository link, triggering a deep semantic audit. It scans for hardcoded API keys, insecure Edge machine learning model loaders (like Python pickle exploits), and prompt injection vectors.
  3. Interactive GitLab Duo Chat: A specialized security assistant trained to suggest high-grade, production-quality remediations. If a vulnerability is selected, Duo automatically targets the code block and provides immediate refactoring solutions.

How we built it

To adhere strictly to high-performance, containerized deployment standards, we engineered Phoneme Orbit with a full-stack architecture:

  • Frontend: React 18 + Vite styled with customized, high-contrast Slate glassmorphism in Tailwind CSS and structural transitions powered by motion.
  • Audio Synthesis Engine: Modeled glottal vocal tract excitation on the Web Audio API using parallel resonant bandpass filters to route a sawtooth glottal pulse.
  • Backend Server: Built a custom Express.js server in TypeScript, bundled to CJS with esbuild for speed.
  • LLM Co-Pilot: Integrated the advanced @google/genai SDK on the server-side to coordinate GitLab Duo and the repository scanner, keeping sensitive API credentials secure.

Mathematical Formulation of Formant Synthesis

To cleanly model human speech phonology without streaming bulky audio assets, we simulated oral filter transfer functions. The acoustics of vowel formants are represented as a sum of exponentially decaying sine waves:

$$ s(t) = \sum_{i=1}^{n} A_i \cdot e^{-\sigma_i t} \sin(2\pi F_i t + \phi_i) $$

Where:

  • $F_1$ and $F_2$ represent the resonant formant frequencies of the vocal tract (modeled under Vocal Throat Opening and Tongue Alignment).
  • $e^{-\sigma_i t}$ acts as the dampening envelope of the vocal resonators.
  • $A_i$ is the relative amplitude boost assigned to the frequency band, keeping high-frequency sibilants and phonological filters balanced.

Challenges we faced

1. Web Audio Q-Factor Attenuation: During initial tests, connecting bandpass filter nodes in series heavily attenuated higher frequencies, resulting in silence when dialing the sliders. I solved this by restructuring the DSP nodes to connect the glottal pulse source in parallel to both formant filters, feeding their synthesized peaks into a combined booster gain node, restoring absolute clarity to the speech synthesizers.

2. Resilience & Cloud Failure Fallbacks: API requests could fail or load-limit under heavy usage. To guarantee continuous service for pupils in classroom environments, I established offline resilience protocols inside the Express backend. If the Gemini API or cloud gateway encounters a 503 error or is offline, system-orchestrated local heuristic rules instantly trigger, providing local, deterministic cybersecurity advice.

Accomplishments that we're proud of

  • Designing a highly custom physical synthesis engine that generates voice mechanics directly in the web browser.
  • Building a secure code scanner from scratch that acts as a real shield against cloud vulnerabilities (RAG prompt injections, pickle file exploits, etc.).
  • Engineering a clean router parameter that lets anyone launch directly into the Phoneme Orbit view with zero intermediate steps.

What we learned

Through Phoneme Orbit, I mastered advanced node-based signal routing in the Web Audio API, standard Express-Vite middlewares, and secure server-side proxying of LLM APIs. Most importantly, it validated that an offline-first strategy is critical for accessibility.

What's next for Phoneme Orbit

The long-term master plan is porting this entire speech synthesis and acoustic logic into SwiftUI and CoreML on Apple Playgrounds. Our ultimate goal is submitting a native application utilizing this architecture to the Apple WWDC 2027 Swift Student Challenge!

Built With

Share this project:

Updates