PAPAGEDON

An agent that watches the crowd with OpenCV 5 and hears the music, then directs live immersive visuals in real time on AWS. It reads the room so every event feels alive.

Comment

Inspiration

I build PAPAGEDON, an immersive music visualizer from Enugu, Nigeria. At every show I saw the same gap: the visuals only hear the music, they never see the room. A VJ reads the crowd and pushes the energy up or pulls it back. I wanted an agent that can do that on its own.

What it does

PAPAGEDON Vision watches a camera feed of the crowd and uses OpenCV 5 to measure two things:

  • Motion, using dense optical flow (how hard the crowd is moving)
  • Density, using background subtraction (how full the room is)

It combines these with live audio from the PAPAGEDON engine into one room energy score:

$$E = \alpha V + (1-\alpha) A$$

Here ( V ) is the crowd energy from vision, ( A ) is the audio energy, and ( \alpha ) adapts: when the crowd and the music disagree, the score eases toward the crowd instead of jumping.

The agent then directs the visuals live, choosing calm or peak scenes from 2D shaders and new 3D Three.js worlds (Neon Tunnel, Particle Galaxy, Audio Terrain, Light Cathedral) to match the room.

  • Auto mode: the agent directs. Manual mode: the VJ takes over with hotkeys (B for Batman, T for Trance, N, G, R and L for the 3D scenes, 1 to 9 for the original shaders).
  • Graceful fallback: with no camera it runs on audio only, and if the cloud service drops it switches to analysis in the browser.

How I built it

  • Vision service: Python with OpenCV 5, in a container on AWS Lambda (pushed through Amazon ECR and exposed by a Lambda Function URL), defined with CloudFormation.
  • Operations: logs and metrics in CloudWatch, files in S3.
  • Frontend: Next.js 15, React 19, WebGL and Three.js at papagedon.com/app, with the Web Audio API for the audio side.
  • Performance: in a 200-frame load test there were 0 errors, a median round trip of 269 ms and a 95th percentile of 290 ms. OpenCV analysis takes about 5 to 7 ms per frame, after a single cold start of about 5.6 s.

Challenges

  • My new AWS account couldn't use CloudFront yet, so I redesigned the setup around a Lambda Function URL for HTTPS.
  • I had to keep the visuals smooth while frames make round trips to the cloud, so the energy is smoothed and scene changes are eased.
  • Lighting in real venues is messy (strobes, darkness), so I tuned background subtraction and flow thresholds to avoid false spikes.

What I learned

Running computer vision in the cloud is very practical: OpenCV 5 analysis takes only a few milliseconds, and most of the delay is the network. Combining several signals beats any single one, because the crowd and the music each catch moments the other misses.

What's next

  • Add PAPAGEDON Vision as a paid endpoint of the AEI (Audio Experience Intelligence) API
  • Multiple cameras and per-zone heatmaps for large venues
  • Pilot it at live events in Nigeria

Built With

Share this project:

Updates

Submission history