Inspiration
A conversation carries meaning through words, facial expressions, gestures, and pauses. When people speak different languages, understanding those layers can take extra effort.
The Babel explores a question: What if a conversation could become a shared work of art?
Inspired by the story of Babel and our experience communicating across languages, we wanted to give people more ways to participate in a conversation. The Babel brings words, gestures, and facial expressions into a shared visual space, where an interaction can leave a lasting trace.
What it does
The Babel is an interactive browser experience that brings multilingual captions, expressive visuals, and generative artwork into one conversation.
Participants appear as animated, camera-colored mosaic particles. Facial actions trigger expressive emoji and shifting geometric compositions, while fingertip movements draw trails across the screen. Five visual families give the artwork different colors, shapes, and rhythms.
Users can explore individually or connect two devices. Each participant independently selects Chinese, English, Japanese, or Spanish. Speech appears as live captions, and the other participant receives a translation in their chosen language. Captions update as participants speak, with typography that adapts to the length of each utterance.
At the end of a session, The Babel generates a conversation portrait from the participants’ recorded facial-action patterns, alongside expression statistics and a facial-activity timeline. Participants can download the artwork as a visual keepsake of their interaction.
How we built it
We built the interface with JavaScript, HTML, and CSS, using Three.js and WebGL for the animated artwork. Browser-based computer vision supplies person segmentation, facial-action signals, and hand tracking.
Deepgram provides streaming speech recognition, and a server-side language-model pipeline generates translations. WebRTC connects the two participants, sharing processed video and interaction data. The application is deployed on Vercel, with its source code hosted on GitHub.
We process facial-action signals using temporal smoothing and validity checks, then aggregate expression occurrences, duration, intensity, and variation over time. Missing observations remain explicitly marked rather than being counted as neutral expressions.
These features drive the final composition: visual-family proportions influence covered area, intensity controls size and lightness, and variation affects rotation. A session-based seed makes the composition reproducible from the same summary data.
When both participants have valid data, each receives an equal total symbol area in the foreground, giving both contributions a visible place in the shared portrait.
Challenges we ran into
Keeping the experience responsive required coordinating camera frames, tracking results, animation, speech recognition, and network updates.
We worked on stabilizing the person silhouette, preserving gesture state through brief tracking gaps, and smoothing transitions between visual states. We also extended the visibility of expressive graphics so that brief facial actions would remain perceptible as participants continued speaking. This visual persistence is separate from the underlying expression statistics.
For translation, each device needed to retain its own recognition language while sending translations in the partner’s preferred language. We refined language switching, handled updates from provisional to final results, and adjusted subtitle placement across both participant views.
Recording a session also required careful timing. We separated total session duration from valid observation time and preserved missing intervals, allowing the results to communicate the limits of what the system observed.
Accomplishments and what we learned
We brought interactive graphics, multilingual captions, two-device networking, and generative artwork into one working experience—from entering a conversation to downloading its visual portrait.
Our biggest lesson was that timing and visual hierarchy matter as much as recognition. A brief expression needs enough time to register visually, and translated text needs a clear place within the conversation.
We also learned that data mapping is a design decision. Choices about area, scale, color, and missing observations shape how participants understand the final artwork. Giving both participants equal foreground space became one way to express our goal of inclusive participation.
Finally, we learned to distinguish observable facial movement from inferred emotion. Our graphics are expressive interpretations of visible actions; they do not establish what someone feels.
Responsible design
Camera analysis runs locally in the browser. Audio is sent to the speech-recognition provider, and text is sent to the translation service. Our application does not persist audio recordings or transcript logs; provider-side processing and retention are subject to their respective policies.
Connected participants receive processed video, which includes the visible camera background, and captions. The current connection does not transmit microphone audio to the other participant.
Facial expressions can carry different meanings across individuals and cultures. We treat our visual mappings as artistic interpretations and leave participants free to explain their own feelings. Missing tracking data is identified explicitly, and the resulting statistics describe observed facial activity.
What’s next
We want to test The Babel with speakers from different linguistic and cultural backgrounds, studying whether the experience helps them initiate conversation, follow each other’s meaning, and feel comfortable participating.
We also plan to improve translation latency and tracking stability, and expand accessibility through adjustable caption timing and visual-motion controls.
Our goal is to make room for different ways of expressing ourselves—and give people a shared artwork to carry forward from the encounter.
Log in or sign up for Devpost to join the conversation.