We created a shared art canvas that uses AI and embeddings to turn people’s words, voices, and gestures into personalized brushstrokes, creating a collective memory of an event that everyone can shape and look back on, together.
Inspiration
Interactive media often centers on one person, and photos or videos don’t let us actively create something together.
We wanted to give people a more engaging way to capture shared moments through art, where everyone contributes to a piece they can keep. We imagined this in venues, public spaces, and events, creating new ways for people to connect with each other and for artists to connect with their fans.
That led us to ask: how can AI bring people together through collaborative experiences, rather than isolate us?
What it does
Momento invites people to respond to a prompt related to the space or event they’re at using their words, tone of voice, and body language.
These signals are mapped to a personalized brushstroke inspired by painters whose styles fit their expression, allowing each person to leave a different mark on the same shared scene. The painting grows throughout the event as more people contribute. At the end, a montage replays the experience stroke by stroke and closes with a short reflection on what everyone created together.
The final artwork becomes a shared memento of the people who were there and how their individual contributions came together across time.
How we built it
Capturing expression We built Momento around a browser experience that captures each visitor’s words, voice, and gestures. MediaPipe Hand and Pose Landmarkers track fingertips, fists, and wrist movement on-device. During the speaking window, we score movement using Laban’s effort qualities: weight, time, space, and flow.
The Web Speech API captures words, while Web Audio measures loudness.
Presence --> paintings
We built a searchable library of 2,656 WikiArt paintings using ArtEmis annotations of how real people felt about each artwork.
A trained emotion model using CLIP text embeddings is blended equally with Gemini’s interpretation, with voice and movement adjusting the emotional read.
We also trained a contrastive text-to-image space to recognize subjects, so “first snow” can find winter scenes as well as paintings that match the feeling.
The combined feeling and subject scores select three paintings and determine their weights, running locally in about 50 milliseconds after Gemini responds.
Creating the shared painting
The painting runs on an NVIDIA L4 GPU on Google Cloud using SDXL-Turbo with four generation steps. IP-Adapter blends the three selected paintings’ image embeddings to guide palette and brushwork. Additional directions adjust energy, warmth, and light. Style-only conditioning limits the influence of the reference paintings’ figures, while Canny ControlNet helps preserve the shared scene’s layout.
The browser reveals the generated painting along the visitor’s drawing path through a ragged, bristle-shaped mask.
Writing and keeping it running
Gemini on Vertex AI handles art direction, event-setup drafts, individual written lines, and the closing reflection. We use a lighter model for writing, with safety checks and fixed fallbacks.
A Python booth server saves each visit’s strokes, generated frame, and canvas so the experience can recover after a crash or reload.
Cached frames, palette washes, and rule-based responses keep the experience running when services are unavailable. A single startup script connects the GPU, tunnel, models, and booth.
Challenges we ran into
- Text alone doesn't carry feeling into art. Turning moods into scene words only changed what was depicted (a moon, snow), not how it was painted
- Text directions like "agitated vs calm brushwork" barely moved the model (directions built from groups of real paintings did: that's why the core is an art-history embedding space)
-Color words, palette matching and recoloring didn't help
- Letting the mood set the light and weather (night with one candle-lit window, spring turquoise sky, rain, snow) did
- Embeddings collapse. Plain CLIP sent 25 of 40 feelings to The Scream; a trained space matched subject instead of feeling (grandmother → portraits of old women); quiet and loud "I'm fine" still land on the same painting
- Paintings leak their content. At full strength, anger brought a brawl of figures into the chapel; style-only injection fixed it
- The art space only reaches where its paintings live. Anger rendered bright yellow until Goya, Delacroix and Géricault joined the library
- Averages turned into mud. A mix of paintings sits near the center, so grief, stress and anger came out muted; pushing the mix outward and adding intensity axes brought them back (+36% visual spread)
- GPU supply was cooked..L4s were unavailable for a while, and we couldn’t run without them
- Latency. The writing model took 8–20 s per line; a lighter model brought it inside the time the painting takes to render
- Tone. Early lines were sappy; we wrote the voice like a museum wall label and kept it to one sentence
- Live-booth reality. Bystanders' hands, a visitor walking away mid-visit, a dead mic or a dropped GPU all had to fail gracefully
Accomplishments that we're proud of
- The painting truly reflects people’s presence. Two visitors with different moods leave visibly different marks!
- Embedding search is fast enough (about 50 ms) to sit inside a live interaction.
- we had a vision and executed on it
- We tend to think of AI’s ability to personalize, but we were able to use it for a collective experience/task
- Real embedding math on real art: the visitor is a weighted point among actual paintings, and the painting moves through art history with the crowd
- Built to survive a live event: every model call has a fallback, and every visit is saved
What we learned
- AI felt collective rather than isolating once each person's input stayed visible inside something shared
- Feeling lives in form (brushwork, palette, edges, light), not in subject matter
- Embedding spaces built from real artworks are more controllable than prompts
- Test the riskiest assumption first: a blind board of rendered images told us more than palette metrics did
- Embeddings collapse onto a few hub paintings unless you measure for it (distinct picks, hubness, paraphrase stability)
- Human-labeled data (how people felt about each painting) beat clever prompting of a general model
- For live experiences, latency and supply (GPU stock, model speed) matter as much as model quality
- Quick experimentation to validate hypotheses are incredibly important, and testing ideas is cheap- set up comparisons/evals to understand what to pursue,
What's next for Momento
- pilots for shared experiences (class, hackathon, museum, concerts, basketball games, etc.)
- An image or gif that gets sent to people as a memory keepsake (NFTs, anyone?)
- building an even richer and nuanced embedding space to create even more faithful representations of people’s embodiment/presence.
- A gallery of works to archive and hold as memories
- RLHF/tuning as we understand if people think their brushstrokes capture what they’re feeling well or not.
