MonaListen

The artist edits polygons in-editor, drawing the river's spline region and configuring its audio settings!
The viewer uses their phone, scanning the surroundings for an immersive audio experience.
The library, where you can choose between your interactive artworks.

Inspiration

Visual art exists for the eye.

But for 295 million people worldwide, that experience is out of reach due to severe vision impairment.

From the Louvre to the Tate Modern, blind visitors drift helplessly through art galleries: dragged around by companions, unable to trace the winding contours of a Renaissance fresco for themselves. Art galleries have historically offered almost no services for blind visitors. After all, how could a blind viewer experience a visual-medium artwork... or can they?

What MonaListen does

Picture a landscape painting of a stream at the foot of a mountain. MonaListen gives artists a visual editor to draw regions, attach generated sound effects, and see live edits to falloff curves and blend radius. Using a live link, a blind visitor points their phone at the painting, moves their hand up/down in space, and hears each sound grow in volume within the mix as the corresponding region is approached.

A visual artist uploads images of their artwork, then uses an editor to place point emitters or draw spline-bounded audio regions across the piece. For each region, they write a text prompt to generate a looping sound effect tied to that part of the painting: from the steady rush of the river to the turbulent wind at the mountaintop. As a music producer myself, I designed the sound controls with care. Artists can spatially configure how each sound blends with distance, choose a falloff curve, and pan either left/right with a simple slider. The finished soundscape is saved with a live link for viewers to use.

A friend helps the blind viewer open the shared link on their phone. Spoken instructions guide their hand until the phone camera intersects the painting, entering the sound landscape the artist built: with a pair of earbuds on, a blind viewer simply moves their phone across the artwork to traverse its coordinate space: exploring the sounds of a living, spatial scene entirely by ear.

How I built it

MonaListen separates artist authoring from visitor playback. The artist-facing React and TypeScript app uses Tailwind CSS and Radix UI for the studio and spline-based region editor. Clerk handles artist authentication; a Node.js and Express backend coordinates uploads, AI requests, and publishing, with Zod validating data at the API boundary. Gemini analysis is optional and artist-initiated. It proposes editable region and segmentation drafts; the artist reviews them before they can be published. ElevenLabs generates the region sound loops and spoken visitor instructions. Artists can preview and approve a loop, and publishing reuses that approved audio. For playback, MindAR tracks the artwork, and Three.js maps the camera pose into the painting’s coordinate space. The Web Audio API mixes the region loops according to the visitor’s tracked position.

For region $i$, its gain can be described as:

$$ g_i(\mathbf{p}) = A_i\,f_i\left(\frac{d_i(\mathbf{p})}{R_i}\right), \qquad y(t,\mathbf{p}) = \sum_{i=1}^{N} g_i(\mathbf{p})\,s_i(t). $$

Here, $\mathbf{p}$ is the tracked position, $d_i$ is its distance from the region’s sound position or geometry, $R_i$ is the artist-set reach radius, $A_i$ is the region’s level, and $f_i$ is its selected falloff curve. Each loop remains an independent layer, so overlapping regions blend instead of cutting abruptly from one sound to another; artist-defined panning is applied per region. Tiger Cloud PostgreSQL with PostGIS stores artwork-region geometry. Replit App Storage holds uploaded images and audio. Publishing packages the reviewed artwork and sound configuration into a versioned listening experience. The core architectural choice was to keep suggestions, spatial mapping, and sound generation distinct: AI can speed up authoring, but the artist decides what is published, and the tracked position (not a sequence of triggers) drives the mix.

Challenges I ran into

Gemini’s suggested outlines often needed adjustment to fit the artwork. I kept the artist in control by making AI-generated regions editable drafts that must be reviewed before publication. I also learned that quota limits, restricted API keys, and invalid credentials can produce similar symptoms, so reliable troubleshooting required distinguishing provider errors rather than reporting every failure as a bad key.

Maintaining continuous sound as the camera moved took more tuning than expected. Abrupt gain changes made the experience feel broken, so I iterated on distance-based mixing to let nearby regions blend naturally. I also had to carefully align the editor’s region geometry with the viewer’s camera-tracking coordinates; even a small offset made the visual indicator feel disconnected from the sound.

Accomplishments that I'm proud of

I built an end-to-end workflow with Replit: artists can upload a painting, map spline-based sound regions, generate and approve audio with ElevenLabs, then publish a versioned listening experience. Gemini can suggest regions and prompts, but the artist can inspect and change every suggestion. The browser-based studio supports laptop use, while the visitor experience is designed for iPhone camera tracking.

I’m especially proud that the regions behave as a spatial mix rather than a sequence of sound triggers. As the visitor moves, sounds can blend according to each region’s settings and distance. I also included spoken guidance, artwork descriptions, VoiceOver support, and helper controls so the experience can be set up for blind visitors.

What I learned

Generative AI is most useful here as an assistant, not an authority: it can reduce setup work, but artists need a clear way to review and correct its output. I also learned that spatial audio quality depends on more than the sound itself: geometry, camera alignment, falloff, and gain transitions all shape what the listener experiences.

Most importantly, accessibility affects the whole interaction model. It needs to inform setup, navigation, feedback, and recovery from tracking problems: not be added at the end.

What's next for MonaListen

The next step is testing with artists and blind and low-vision visitors in real gallery conditions. I want to evaluate tracking across devices, lighting, and viewing distances, and learn whether the spoken guidance and sound transitions work as intended in front of actual paintings.

From there, I’d refine onboarding and region-authoring tools based on feedback, strengthen recovery when tracking is lost, and continue validating VoiceOver and other accessible interactions. Any visitor analytics should remain off by default, consistent with the project’s privacy-first approach.