Vision Sync

Vision Sync — Real-Time Audiovisual Art Engine

Comment

Inspiration

I wanted to explore what happens when you bridge real-time computer vision with generative ambient music and futuristic canvas art. The concept was inspired by digital synesthesia—giving users the ability to perform, manipulate sound, and generate HUD-style artwork simply using hand gestures, face expressions, and physical objects in front of their camera.

What it does

Vision Sync turns a standard webcam into a real-time reactive audiovisual canvas:

  • Groq Llama 3.3 70B & Gemini AI: Provides ~14ms spatial intelligence, dynamic scale recommendations, and atmospheric visual suggestions.
  • Futuristic Hand Rendering Engine: Displays sci-fi holographic corner brackets, rotating palm reticles, and plasma arcs in themes like Cyberpunk Neon, Hologram Blue, Astral Gold, and Matrix Emerald.
  • Gesture Emotes & Flourishes: Hand poses (Peace ✌️, Thumbs Up 👍, Heart 🫶, Fist ✊, Pointing 👆) trigger particle explosions, canvas interactions, and sound arpeggios.
  • 3D Facial Expression Biometrics: Tracks 468 3D facial landmarks to modulate audio scales, octave pitches, and ambient synth harmonics.
  • Physical Object Acoustic Resonance: Detects everyday physical items (phones, coffee cups, books, plants) using ML to dynamically shift ambient musical genres.

How I built it

  • Frontend Core: Built with React 19, TypeScript, Vite, Tailwind CSS v4, and Motion.
  • Computer Vision & ML: Integrated MediaPipe Tasks Vision (FaceLandmarker, HandLandmarker) and TensorFlow.js (coco-ssd) for decoupled object detection.
  • Generative AI: Powered by Groq Llama 3.3 70B Versatile for ultra-fast spatial reasoning and Google's Gemini Flash.
  • Audio Engine: Synthesized in real-time using Web Audio API and Tone.js.

Challenges I ran into

Maintaining a solid 60 FPS performance without main-thread UI lag while running multiple heavy neural network models simultaneously was a massive hurdle.

I tackled this by:

  1. Running MediaPipe directly on requestAnimationFrame using WebGL/WASM delegates (~2ms execution time).
  2. Decoupling COCO-SSD object detection into non-blocking background promises triggered every 1200ms.
  3. Implementing dynamic Adaptive Velocity LERP (0.55 to 0.92) to smooth jitter while keeping fast movements instant.
  4. Replacing heavy canvas shadowBlur operations with a custom native double-stroke vector glow, yielding a 4x FPS boost.

Accomplishments that I am proud of

  • Achieving a native, silky-smooth 60 FPS frame rate on standard hardware with zero input lag.
  • Seamlessly blending multiple AI/ML systems (spatial LLM reasoning, face mesh, hand tracking, object recognition) with dynamic Web Audio synthesis.
  • Designing high-tech sci-fi visual overlays that react organically to human gestures and facial biometrics.

What I learned

  • Techniques for optimizing heavy WebGL and WASM workloads on the browser main thread.
  • How to tune dynamic math interpolation (LERP) algorithms for ultra-responsive hand-tracking UI elements.
  • Practical implementation of real-time audio synthesis modulated by dynamic computer vision landmark coordinates.

What's next for Vision Sync

  • Multi-user Collaborative Sync: Allowing multiple people in different webcam streams to perform ambient music together.
  • Expanded Sound Engines: Integrating custom MIDI export capabilities so users can export their camera performances directly into DAWs.
  • Custom 3D Particle Shaders: Adding WebGL dynamic shader canvases for even richer visual FX.

Built With

Share this project:

Updates