Speech Constellation

Nobody has ever seen the shape of their own voice. Speech Constellation draws it live, in your browser, from nothing but a microphone.

Comment 1

Inspiration

Most entries in an art and technology competition follow the same pattern: a prompt goes in, an image comes out, and the computer functions as a faster paintbrush. I wanted to answer the theme literally instead. If the brief is art that could not exist without technology, then the honest test is whether the subject of the work is something a person could perceive at all without a machine.

That test led me to the human voice, and specifically to a fact I found striking: nobody has ever seen the shape of their own voice. The spectral shape of speech exists for roughly twenty milliseconds at a time. It cannot be painted, because it cannot be seen. It cannot be photographed, because it is not light. The only route to looking at it is for a machine to measure it thousands of times per second and give it a body inside a space that has no physical equivalent.

The second idea came from birdsong visualisers, which reuse a position when a bird repeats a phrase rather than drawing a new mark. Applied to human speech, that produces something more interesting than a waveform. It produces a picture of a person's habits: which sounds they return to, and how much of their speaking is repetition.

## What it does

Speech Constellation analyses the microphone sixty times per second and reduces each frame to four numbers describing the shape of the sound. Those numbers position a point inside a slowly rotating cube, where it becomes a star.

| Axis | Meaning | |---|---| | X, centroid | Brightness, meaning where the spectrum's energy sits. A hum is dark, an "sss" is bright. | | Y, excess width | How much wider or narrower the sound spreads than its brightness alone predicts. | | Z, excess purity | How much more or less tonal the sound is than expected. | | Colour | Pitch, estimated by autocorrelation. |

The central mechanic is recurrence. A frame that lands near an existing star does not create a second star. It revisits the first one, which gains weight, brightens, and thickens the path walked to reach it. Repeating a sound re-lights the constellation rather than obscuring it with duplicates.

Four further features build on that:

  • A guided introduction of four prompts (hum, hiss, say your name twice, count to five twice) teaches the space by asking the visitor to make the sounds and watch where they land. Each prompt leaves a labelled landmark, so the cube ends up annotated with the visitor's own voice.
  • The constellation sings. A drone follows its centre of mass, and every recurrence rings a bell tuned to that star's pitch and panned to its position. Nothing sounds when a new star is created, so the piece only sings when the speaker repeats themselves.
  • One control exports a 1080x1350 portrait of the constellation with the session's statistics and a name derived from them.
  • Audio never leaves the device. Nothing is recorded and nothing is uploaded.

## How we built it

The project has zero runtime dependencies. There is no rendering library, no audio library, and no framework. Everything that ships is in src/, built from the Web Audio API and Canvas 2D.

  audio.js       microphone capture, RMS level, autocorrelation pitch detection
  spectral.js    FFT to centroid, spread, flux and crest
  trajectory.js  the recurrence graph, axis fitting and auto-calibration
  projection.js  hand-written 3D to 2D projection
  render3d.js    Canvas 2D drawing: glow sprites, depth cueing, comet trail
  sonify.js      Web Audio synthesis: drone and pentatonic recurrence bells
  ritual.js      the four-step guided introduction
  poster.js      the voiceprint export
  demovoice.js   a synthetic voice, so the piece is never empty
  main.js        application state and wiring

An AnalyserNode supplies both a time-domain window, used for RMS level and for autocorrelation pitch detection with an octave guard and parabolic peak refinement, and a magnitude spectrum, from which spectral.js computes the spectral centroid:

$$ C = \frac{\sum_k f_k m_k}{\sum_k m_k} $$

along with spectral spread, flux and crest. Those features feed a recurrence graph, and a hand-written projection places each node on screen by rotating it, tilting it, and dividing by distance. Rendering avoids shadowBlur in favour of pre-rendered glow sprites stamped in additive mode, because at several hundred points per frame the former stalls.

## Challenges we ran into

The axes were carrying duplicate information. Spectral spread is highly predictable from the centroid, since bright sounds are also wide. I measured the correlation at \( r = +0.95 \), which meant the second axis was very nearly a copy of the first, and the constellation collapsed into a diagonal ribbon inside an otherwise empty cube.

My first instinct was to divide spread by centroid. Measuring that before shipping it showed it was wrong: it overcorrects to \( r = -0.93 \) and draws the same ribbon with the opposite slope. Both are special cases of one general form, so the fix was to measure the relationship rather than assume it:

$$ Y = \log S - \left( K \log C + B \right) $$

where \( K \) and \( B \) are fitted by ordinary least squares from the same samples that calibrate the axis ranges. The fit converges near \( K \approx 0.66 \), giving \( r = -0.23 \).

Fixing one axis pair revealed another. The cloud still sat on a plane, because I had never addressed the third axis. Crest correlates with centroid at \( r = -0.96 \), since vowels are dark and tonal while fricatives are bright and noisy. Applying the same residual treatment to the crest and flux axes raised the number of distinct volume cells occupied from 82 to 109 and dropped the strongest correlation between any pair of axes from \( -0.86 \) to \( -0.20 \).

Calibration deadlocked. The axis ranges were originally fitted from the merged graph nodes, and that could never converge. The starting ranges are far wider than a real voice covers, so at those ranges the merge radius absorbed every vowel into roughly three nodes, and the node count therefore never reached the threshold that would have narrowed the ranges. Fitting from raw frames instead, retained by reservoir sampling, removes the feedback loop.

One exception could black out the artwork permanently. A negative age produced a negative circle radius, arc() threw, and the requestAnimationFrame loop died for the rest of the session. I clamped the cause and also wrapped the draw call, so that a single bad frame is reported once and the next frame is still attempted.

An accessibility bug hid in plain sight. The introduction panel sets display: grid, which outranks the browser's own [hidden] rule, so setting hidden never actually removed it. It only appeared to be dismissed because the fade-out animation left it at zero opacity. Since I disable that animation under prefers-reduced-motion, any visitor with that preference enabled could not dismiss the opening screen at all. I found it only because a screenshot showed the panel still present after it had supposedly been hidden.

Judges may not have a working microphone. Free demos fail in exactly this way, so I built a synthetic voice model that speaks into the same graph and is always labelled as a demonstration. It emits spectral features directly rather than playing audio, because browsers keep an AudioContext suspended until a user gesture, which is precisely the moment the screen must not be blank.

## Accomplishments that we're proud of

The piece has no runtime dependencies at all. The 3D projection, the spectral analysis, the glow rendering, the recurrence graph and the audio synthesis were each written from their definitions rather than imported.

The design decisions were measured rather than argued. When I suspected the axes were redundant I wrote a benchmark, computed correlations and volume occupancy on a fixed test voice, and let the numbers choose. That process also produced a result I would not have guessed: forcing all three axis correlations to exactly zero measures worse, not better, at 27 percent of the cube's volume against 46 percent. Perfect orthogonality compresses the residuals into a tight ball in the middle, whereas leaving some structure lets sounds separate into distinguishable clusters. What ships is the arrangement that measured best, not the one that is mathematically purest.

Finally, the work degrades gracefully. A denied microphone, a missing microphone, a locked-down machine, or a browser that refuses to produce sound all leave a running artwork rather than an error.

## What we learned

Measuring an assumption is cheap, and shipping one is expensive. The instinct to divide spread by centroid felt obviously right and was demonstrably wrong, and a ten-minute benchmark was the difference between fixing the problem and inverting it.

Correlated inputs waste dimensions. If two axes carry the same information, the third dimension of a 3D visualisation is decorative. The general lesson is that a residual is often more informative than a raw measurement, because it shows only what has not already been said.

third dimension of a 3D visualisation is decorative. The general lesson is that a residual is often more informative than a raw measurement, because it shows only what has not already been said.

Reliability is a design decision, not an operational afterthought. Asking what a visitor sees when the microphone fails changed the shape of the project, and produced the demonstration mode that turned out to be the most reliable part of it.

## What's next for Speech Constellation

  • A shareable link that encodes a constellation into the URL, so that one person can open another person's session.
  • Session replay, showing a constellation redrawing itself from nothing.
  • A two-voice mode that colours stars by speaker, revealing where two people's speech overlaps.
  • Wider browser and device verification, since development and testing so far were carried out in Chrome on desktop.

Built With

Share this project:

Updates