Echo Canvas — Paint With Your Voice and Face

Inspiration

Nearly every creative-coding tool assumes the same thing: a hand on a mouse, a finger on a touchscreen, a stylus on a tablet. For someone with a mobility impairment that can't hold a brush, that assumption is the barrier — not talent, not creativity, just the interface. We wanted to build a painting tool where the interface disappears entirely and the art comes directly from what a person can always do: speak, make sound, and express something on their face.

Echo Canvas isn't "assistive technology art" as an afterthought — it's built on the idea that voice and expression are a richer creative input than a mouse ever was, because they carry emotion the cursor never could.

What it does

Echo Canvas is a full-screen, real-time generative painting studio driven entirely by your voice, vocal tone, spoken words, and facial expression.

A p5.js particle/flow-field canvas fills the screen, continuously evolving.
The Web Speech API transcribes what you say and scans it against a keyword-sentiment lexicon — words like calm, fire, chaos, ocean shift the emotional target the canvas is painting toward.
The Web Audio API analyzes your live mic input for volume (RMS) and pitch/spectral content — loud, high-energy speech pushes particles outward and speeds them up; quiet, low-pitched speech thickens stroke weight and slows everything down.
face-api.js tracks your expression through a small, toggleable webcam preview — smiling brightens the palette and tightens particles into harmonic spirals; surprise/mouth-opening triggers dynamic particle bursts.
All three channels fuse into a single Emotional Index — a point on a valence–arousal plane — that continuously drives color, particle speed, spiral tightness, and stroke behavior. Loud + smiling produces explosive gold harmonic spirals; quiet + sad produces slow, dripping deep-blue watercolor textures.
"Finish Session" plays back the last 10 seconds of the artwork's emotional evolution as an accelerated loop, auto-generates a poetic caption describing the emotional arc (e.g. "A moment of stormy energy transformed into calm focus"), and exports a one-click memory card (PNG + mood-mix bars + JSON metadata).

How we built it

Signal extraction. Three independent sensing modules run in parallel: face-api.js's expression classifier scores happiness/surprise/sadness from the webcam feed; a Web Audio AnalyserNode extracts RMS volume and spectral pitch from the mic in real time; and a keyword lexicon scores the live Speech Recognition transcript for valence/arousal content.

Fusion. Rather than letting each channel trigger its own independent effect (which tends to look like three unrelated animations layered on top of each other), we fuse all three into one valence–arousal point using a weighted sum:

$$ V = \alpha V_{\text{face}} + \beta V_{\text{voice}} + \gamma V_{\text{word}}, \qquad A = \alpha A_{\text{face}} + \beta A_{\text{voice}} + \gamma A_{\text{word}} $$

with default weights $\alpha{=}0.4,\ \beta{=}0.35,\ \gamma{=}0.25$ (face weighted slightly higher, since it's the most continuous and least noisy signal). The result is passed through an exponential moving average,

$$ E_t = E_{t-1} + \lambda\,(E_{\text{target}} - E_{t-1}), \qquad \lambda \approx 0.08, $$

so a single blink or word doesn't cause the canvas to jerk — the artwork's emotional state moves the way a real mood does, gradually.

Mapping to visuals. The smoothed $(V,A)$ point is converted to polar form: angle $\theta = \operatorname{atan2}(A,V)$ selects hue around a circumplex-style color wheel, and magnitude $r = \sqrt{V^2+A^2}$ scales particle speed, spiral tightness, and stroke turbulence. Because everything derives from one continuous point, transitions between emotional states are smooth blends, not scene swaps.

Timelapse & memory card. The fused state is snapshotted every 500ms. On "Finish Session," those keyframes replay as a fast 10-second loop, and a small template-based caption generator reads the trajectory's start/end quadrants to describe the emotional arc in natural language, before exporting a combined PNG + JSON memory card.

Build. Vanilla JS + p5.js for rendering, face-api.js for expression inference, native browser Speech Recognition and Web Audio APIs for sensing — no native app, no install, built and deployed on Lovable.

Challenges we ran into

Getting three asynchronous, differently-noisy input streams — a discrete, sometimes-empty speech transcript; a continuous but jittery audio signal; and a classifier that occasionally flickers between expressions — to feel like one coherent emotional signal instead of three animations arguing with each other was the core engineering challenge. The fix wasn't more effects, it was fewer: collapsing everything into a single valence-arousal coordinate with proper smoothing, so the canvas always has exactly one thing to express at a time.

Pitch extraction from raw mic input in-browser is also noisier than it looks on paper — we had to lean on spectral centroid as a proxy rather than true fundamental-frequency detection to keep it stable enough to drive visuals without constant flicker.

Designing for the accessibility goal meant every control had to work with sensing alone eventually turned into a genuine constraint we respected throughout: the webcam preview had to be fully optional without disabling the tracking behind it, since forcing self-view isn't acceptable for every user.

Accomplishments that we're proud of

A genuine multi-modal fusion model — not three independent effects, but one mathematically coherent emotional signal driving every visual parameter.
A painting tool that requires zero pointer input from start to finish, built with the specific goal of being usable by someone who cannot hold a brush.
The timelapse + auto-captioned memory card, which turns a session into something shareable and legible to someone who wasn't in the room — the caption alone tells you the emotional story of the piece.
Shipping all of this — webcam inference, audio analysis, speech recognition, and generative rendering — running smoothly together, client-side, in a browser tab.

What we learned

That fusing signals mathematically (one shared valence-arousal coordinate) produces a far more convincing "the art understands me" feeling than triggering separate effects per input ever could — coherence mattered more than the number of features. We also learned how much smoothing (the EMA constant $\lambda$) is doing the emotional work: too fast and the canvas feels twitchy and mechanical, too slow and it feels unresponsive; getting that one constant right changed the entire character of the piece.

What's next for Echo Canvas

Per-user calibration — a short onboarding that learns an individual's baseline expression and vocal range, since "smiling" and "loud" vary a lot person to person.
Multi-session emotional diary — stringing memory cards together over time into a longer visual/emotional history.
Switch/eye-gaze input option — for users for whom voice or facial movement is also limited, adding a third accessible input path into the same fusion engine.
Collaborative canvas — multiple people's Emotional Index streams blended into one shared artwork in real time.