Inspiration
Joy Of Painting began with a simple, happy idea: what if painting along with Bob Ross could become a friendly paint along between a person and an AI agent?
Bob taught us that painting is not about perfection. It is about slowing down, trying something new, and turning little accidents into trees, clouds, and possibilities. We wanted to bring that peaceful philosophy into an interactive studio where humans and agents can learn from the same lesson while creating entirely different paintings.
What it does
Joy Of Painting is a collaborative watercolor studio built around The Joy of Painting.
The experience includes:
- Separate YouTube players for the human and AI agent
- A synchronized live transcript that follows the current lesson
- Resizable video players and painting easels
- A WebGL watercolor canvas with wet diffusion, pigment flow, paper texture, and drying
- Bob Ross–inspired brushes, including flat, fan, round, liner, blender, and painting-knife tools
- Physical pigment mixing based on the Kubelka–Munk model
- Episode-specific color palettes
- An eyedropper for collecting colors from the lesson
- WebMCP tools that let an agent control its video, read captions, inspect the lesson, and paint on its own canvas
Every artist receives an easel. One belongs to the human, and one belongs to the agent. Then we let the happy little competition begin.
How we built it
We built the studio with React and Vite, using Video.js’s React YouTube integration for the two independent lesson players.
The watercolor surface is a real HTML <canvas> powered by WebGL2. Its simulation models surface water, suspended pigment, deposited pigment, capillary movement, evaporation, paper fibers, granulation, and staining. Instead of averaging RGB values, colors are blended using Kubelka–Munk absorption and scattering calculations to behave more like physical paint.
The transcript system maps timestamped captions to the active episode and updates the central caption as the lesson progresses.
For agent interaction, the application registers browser-native WebMCP tools through document.modelContext. These tools allow an agent to:
- inspect the lesson and canvas
- seek, play, or pause its own YouTube player
- read the current transcript
- request a lesson reference image
- paint normalized watercolor strokes
- dry, undo, or clear its own easel
Finally, we deployed the experience globally using Cloudflare Workers and static assets.
Challenges we ran into
Watercolor does not like to sit still—and neither did our first simulation.
We had to solve canvas resizing feedback loops, animation-frame cleanup, WebGL resource management, and React lifecycle issues without losing the organic movement of wet pigment.
Color preservation was another important challenge. Early versions stored old strokes in the active pigment slot, which meant loading red could turn previously painted yellow strokes red. We fixed this by baking existing pigment into the paper using its original optical properties before loading a new color.
Synchronizing two separate YouTube players with episode changes and live captions also required careful handling. YouTube’s cross-origin iframe security prevents page JavaScript from capturing exact video-frame pixels, so the agent receives an honest timestamped lesson reference rather than a pretend screenshot.
There were plenty of happy little bugs—but each one showed us where another tree could live.
Accomplishments that we're proud of
We are especially proud that the AI agent is not merely commenting on the painting. It can genuinely participate.
During testing, the agent independently moved through a Bob Ross lesson, read the live transcript, and painted a complete woodland composition using only the website’s WebMCP tools.
We are also proud of:
- A genuine WebGL watercolor simulation
- Non-destructive pigment changes that preserve previous colors
- Independent human and agent workspaces
- Live episode-aware captions
- Physical color mixing
- More than 400 browsable Bob Ross lessons
- A calm interface that keeps painting—not technology—at the center
What we learned
We learned that agent-friendly software works best when capabilities are explicit, focused, and honest.
WebMCP allowed us to expose meaningful actions such as “paint a stroke” or “read the current caption” instead of asking the agent to guess its way through screen coordinates.
We also learned that believable digital paint depends on time. Water must spread, pigment must settle, edges must darken, and colors must preserve their history. The smallest details often create the strongest feeling of physicality.
Most importantly, we learned that human–AI collaboration does not always need to be about productivity. Sometimes it can simply be about making something peaceful together.
What’s next for Joy Of Painting
Next, we would like to add:
- Frame-accurate visual references through an authorized video-processing pipeline
- Saved galleries and downloadable side-by-side results
- Timed human-versus-agent painting challenges
- Brush-pressure and tilt support for drawing tablets
- Smarter agents that plan paintings from transcript cues
- Collaborative multiplayer studios
- Automatic judging based on composition, color, and creativity
- More paper surfaces, pigments, drying effects, salt, blooms, and glazing
- A replay mode showing both paintings developing stroke by stroke
There are still many happy little worlds left to paint.
Built With
- chatgpt
- codex
- javascript
- react
Log in or sign up for Devpost to join the conversation.