Chorus

CHORUS turns your webcam and microphone into a paintbrush

Comment

Inspiration Every digital canvas I'd ever used reduced creativity to a mouse click and a color picker — flat, silent, solitary. I wanted to ask a different question: what if the canvas itself could see you and hear you? What if joy actually looked like gold, and anger actually looked like red spikes, because a machine was watching your face decide that in real time? CHORUS started as that one mechanic — emotion as pigment — and once it worked, it was obvious it shouldn't stop at 2D painting alone. If a webcam and a particle system could be a brush, they could also be a sculpting tool, a mirror, a multiplayer canvas, even the source sketch for a generated website.

What it does CHORUS is a suite of connected creative modes sharing one account and one particle-physics core:

Solo — an 11-tool painting canvas (particle brush, hand-draw, 7 traditional pen styles, shapes, lines, voice-paint, selection, more) where your face and hands are always live inputs, and every saved piece gets an AI-written poem. Collective — up to 8 people paint on one canvas from different devices at once, each rendered as their own emotion-driven particle swarm, with a built-in Game Arena (5 mini-games: Pictionary, Impostor, Draw, Memory, Blind). Mirror — a live face-particle reconstruction pushed through 17 real-time camera effects (thermal, hologram, x-ray, glitch...). Sculpt — a full 3D modeling studio in the browser with an audio-reactive performance mode, exporting to GLB/OBJ/STL/PLY/USDZ. WebForge — sketch a wireframe, and Gemini turns it into a real, editable website, preserving the exact colors you drew. Social — a feed, competitions, badges and a leaderboard tying it all together. How I built it The core loop is entirely client-side: MediaPipe's FaceLandmarker/HandLandmarker extract face blendshapes and hand landmarks from the webcam every frame; pure mapper functions turn those into an emotional state ${e_1,\dots,e_6}$ and a gesture, which parametrize a custom p5.js particle-swarm simulation (steering forces + simplex noise) rendered in instance mode so several independent canvases can coexist. Audio is analyzed via Web Audio FFT into bass/mid/treble bands that modulate the same swarm.

Everything else grew around that: React 18 + Vite on the frontend, Three.js/R3F for Sculpt, Fabric.js + Monaco for WebForge, and a dual-backend architecture — a local Node/Express + Socket.io stack for fast iteration, and Supabase (Postgres/Auth/Storage/Realtime) + Vercel serverless functions for the hosted deployment. Every data hook picks its backend automatically depending on whether Supabase is configured, so the same UI code runs in both environments unmodified.

Challenges I ran into The hardest single problem was Collective on a serverless host. Vercel can't hold a persistent WebSocket, so Socket.io — which the whole multiplayer canvas and Game Arena were built against — simply couldn't survive deployment. Rather than redesign every game and UI component, I reverse-engineered the interface my Socket.io hook exposed (rooms, presence, chat, strokes, camera frames, game events) and rebuilt it 1:1 on top of Supabase Realtime's presence and broadcast channels, with a host-authoritative pure-function game engine so only one client ever owns timers and scoring. Because the interface matched exactly, the multiplayer UI needed zero changes when the transport underneath it changed completely.

The other recurring challenge was trustworthy AI generation — keeping Gemini's WebForge output honest to the colors the user actually drew (built a dedicated site-palette extraction/fidelity system instead of trusting the model's own color choices), and handling quota/rate-limit failures gracefully instead of silently breaking the creative flow.

What I learned That "emotion as a rendering parameter" is a surprisingly deep design space once gesture and audio are added on top of it — and that building a contract (a stable public hook interface) before swapping an entire realtime transport underneath a feature is what makes a rewrite like the Socket.io → Supabase Realtime migration safe instead of terrifying.

Built With

Share this project:

Updates