SynthLM

An AI-powered web synthesizer that translates musicians' abstract artistic imagination into explainable, reproducible synthesizer parameters.

Comment

Inspiration

The inspiration behind SynthLM comes from two core challenges I experienced as an arranger and music producer:

  1. Translating abstract artistic vision into sound: As an arranger, composers and producers often describe the sound they want using poetic, abstract metaphors. Because they think like artists, these imaginative descriptions are notoriously difficult to translate into tangible synthesizer patches and practical acoustic designs. I wanted to build a bridge that turns poetic intent into reality.

  2. Lowering the steep barrier of synthesizer sound design: When I first started learning music arrangement, the sheer number of complex synthesizer parameters was intimidating. I often had an intuitive goal in mind—making a patch more ethereal with deeper reverb, or giving it a punchier, shorter attack bite—but I didn't know which specific knobs, envelopes, or cutoff frequencies to tweak to achieve the sound in my head.

SynthLM leverages AI to empower both seasoned arrangers and newcomers to effortlessly reflect the sounds in their imagination directly into their music.

What it does

At its core, SynthLM is a fully playable, zero-latency web synthesizer built with the architectural depth of vintage hardware, featuring:

  • Dual polyphonic oscillators with octave transposition, detune, and level mixing
  • Multi-mode resonant filter (lowpass, highpass, bandpass)
  • Dynamic filter envelope (FLT ENV) specifically calibrated for percussive acoustic attack transients (such as electric piano tines and punchy brass swells)
  • 4-chain studio effects rack featuring Drive, Stereo Chorus, Feedback Delay, and Reverb
  • Versatile LFO modulation matrix routing to pitch vibrato, filter breathing sweeps, and volume tremolo

On top of this acoustic foundation, SynthLM delivers two core innovations:

  1. Natural-language sound design powered by Google Gemini: Whether a musician inputs technical instructions (e.g., "add more reverb" or "shorten attack") or poetic, synesthetic metaphors (e.g., "a solitary Icelandic aurora pad" or "1980s Japanese City Pop electric piano"), Gemini deconstructs the intent into four acoustic pillars: Timbre, Brightness, Dynamics, and Space. It accurately maps the sound to 22 physical parameters while providing educational, physics-grounded explanations for every modified knob—making sound design transparent and accessible even to beginners.

  2. Interactive hand-drawn wave sculpting with 16-harmonic Fourier analysis: Musicians can draw custom single-cycle curves directly on the oscillator canvas. Our engine computes a 16-harmonic Fourier series in real time via Discrete Fourier Transform (DFT), synthesizing smooth, alias-free Web Audio PeriodicWaves on the fly. This provides both the human producer and the AI designer with an expressive palette of bespoke overtone colors far beyond standard sine, square, or sawtooth waves.

How we built it

We built SynthLM with a responsive, instrument-grade web audio architecture:

  • Frontend & Visuals: React 19, TypeScript, and Vite, styled with Tailwind CSS v4. The centerpiece 60 FPS oscilloscope and FFT spectrum analyzer is rendered smoothly on HTML5 Canvas.
  • Audio DSP Engine: Built on Tone.js and native Web Audio & Web MIDI APIs, orchestrating dual oscillators, resonant filters, dynamic filter envelopes, LFO modulation, and a 4-chain studio FX rack.
  • AI Sound Designer: Integrated with Google Gemini (gemini-4-argon and gemini-3.8-flash) via the official @google/genai SDK, enforcing strict JSON Schema formatting.

Challenges we ran into

During the build, we solved two major technical hurdles:

  1. Audio latency and playing responsiveness: For any playable instrument, latency is critical. Initially, when connecting an external MIDI keyboard, there was noticeable input lag that made playing chords feel disconnected. To fix this, we set the Web Audio look-ahead buffer to zero (lookAhead = 0) and optimized parameter ramping, achieving sub-5ms interactive response with immediate, hardware-like tactile feedback.

  2. Grounding abstract AI creativity into real acoustic physics: LLMs can easily hallucinate or output chaotic, unmusical parameter values when handed poetic metaphors like "a solitary winter aurora." To keep the AI grounded, we anchored the system in four acoustic pillars (Timbre, Brightness, Dynamics, Space) and implemented Discrete Fourier Transform (DFT) harmonic math for hand-drawn waveforms. Carefully engineered system instructions ensure Gemini's sound design strictly obeys the laws of acoustic physics.

What we learned

The biggest lesson from building SynthLM is that while people often view AI as omnipotent, it fundamentally isn't. It is easy to assume AI understands everything, yet human emotion and subjective feeling remain an uncharted space for raw language models. In this project, by establishing rigorous physical frameworks, acoustic boundaries, and mathematical grounding, we enabled AI to truly interpret and honor human emotion, translating artistic intuition into physical acoustic reality.

What's next for SynthLM

Looking ahead, we envision SynthLM evolving in two exciting directions:

  1. Packaging into native DAW plugins (VST3 / AU): Bringing this real-time AI sound design experience directly into industry-standard digital audio workstations like Ableton Live, Logic Pro, and FL Studio.
  2. Preset & modulation export for flagship industry synths: Expanding SynthLM's AI translation capabilities to output patch formats and wavetables compatible with renowned synthesizers such as Xfer Serum and Spectrasonics Omnisphere.

Built With

Share this project:

Updates

Submission history