Synapse

Synapse decodes tone, emotion, and intent — making online conversations more humanized.

Comment

Inspiration

In online conversations, emotion often gets lost in translation. A message like "Sure." could mean agreement — or quiet frustration. A simple voice note could sound different to every listener.

We built Synapse to close that gap — translating not just what’s said, but how it’s meant. Like neural synapses that transmit signals between brain cells, Synapse connects emotions between people — bringing empathy back into digital communication.

What it does

Synapse is an AI-powered communication app that understands and expresses human emotion across text, audio, and facial expressions.

Core Features

Multimodal Emotion Detection
Synapse enables users to communicate with emotional depth across voice, text, and facial expressions, blending them into a single emotional language.

Users can interact with Synapse in three natural ways:

  1. Voice Messages — Emotion from Tone
    Record and send an audio message directly within Synapse. Our AI instantly analyzes the tone, pitch, and rhythm of your voice using a speech emotion recognition model. The system then transcribes your message and displays both the text and its detected emotion — so the listener understands what you said and how you felt when saying it.

  2. Text Messages — Emotion from Expression
    When typing in the chat, your webcam subtly captures facial cues in real time. These expressions are interpreted to infer your emotional state as you write. Synapse then attaches the detected emotion to your text message, transforming a simple “Sure.” into a message that carries emotional transparency — whether calm, cheerful, or frustrated.

  3. Emotion-to-Audio Conversion — Expressive Voice Generation
    For any message that already includes emotional annotations — whether written by you or received from another user — Synapse can generate an emotionally expressive voice message. Using instant voice cloning, it converts the text into speech that sounds like you or other users, but with the emotion naturally embedded. This allows others to hear your tone, not just read your words.

Together, these modalities allow Synapse to interpret emotion holistically — uniting text, voice, and expression into one continuous emotional experience.

Emotion Visualization
Each emotion is mapped to a distinct color, creating a dynamic visual spectrum that brings conversations to life and allows users to intuitively sense the emotional flow.

Emotion-Aware Voice Conversion
Users can transform messages into emotionally expressive audio using the ElevenLabs API and instant voice cloning. This allows recipients to hear not only the words, but the intent and emotional nuance behind them. Senders can select or fine-tune the emotional tone before sharing.

Emotion Heatmap (Community Feature)
Messages contribute to a geolocation-based heatmap, visualizing the emotional landscape of a community—whether calm in the library, cheerful near the café, or slightly stressed during exam season. The result is a living map that reflects collective emotions in real time.

Together, Synapse goes beyond a traditional chat app—it forms an emotional network, reconnecting human empathy in digital communication.

How we built it

Our stack combines real-time communication, machine learning, and geospatial visualization.

Frontend

The frontend is built with Next.js, providing a seamless and interactive user experience. It supports real-time chat, dynamic emotion visualizations, and an interactive map interface for the community heatmap. Components are designed for responsiveness and performance, ensuring smooth rendering of text, audio, and video-based emotion annotations.

Backend

The backend is powered by Node.js with Express, orchestrating API endpoints for messaging and emotion data generation. tRPC enables type-safe communication between the frontend and backend, while Prisma handles database interactions with efficiency and reliability. Pusher facilitates real-time updates, ensuring instant delivery of messages, emotion annotations, and heatmap updates across all connected clients. This stack ensures scalability, responsiveness, and robust handling of multimodal emotion data.

Multimodal Emotion Inference Engine

Our inference engine is a custom Flask-based server running on a Runpod container, orchestrated by a Gemma 3 (4B) LLM to form a multi-agent system. Depending on the input type—text, audio, or video frames—the engine coordinates the appropriate models to detect and annotate emotions.

Key components include:

The engine produces emotion-annotated outputs that are either further processed (e.g., for emotion-aware audio generation) or delivered to the frontend for visualization and interaction. HTTP endpoints allow the Node.js backend to interact seamlessly, while Supabase manages larger data transfers such as generated audio.

Audio Generation

Synapse provides emotionally expressive, personalized audio messages by combining the ElevenLabs API with instant voice cloning. Users can clone their own voice, allowing text messages annotated with emotion to be converted into speech that sounds like them.

The workflow is coordinated by the Gemma 3 multi-agent system: emotion-annotated text is refined by the LLM to produce natural, spoken-language phrasing with embedded emotional tags, which guides the TTS engine. When the user triggers audio generation, the system produces the output within ~5 seconds and streams it to the frontend for immediate playback. Users can replay their personalized, emotion-aware audio messages at any time.

Emotion Heatmap

The emotion heatmap leverages Leaflet.js and OpenStreetMap to provide an interactive, real-time visualization of aggregated emotional data. Each message contributes to a geolocation-based map, allowing users to see the collective emotional landscape of a community. The heatmap updates dynamically as new messages arrive, revealing patterns such as calm study zones, bustling social areas, or moments of stress during exams. Designed for clarity and responsiveness, it offers both an overview of community sentiment and the ability to explore localized emotional trends.

Challenges we ran into

There are several challenges we ran into, but the most can be categorized into the following:

  1. We had issues with backend and frontend type consistency.
  2. LLM are sometimes unreliable, so we spend quite a lot of effort on prompt engineering to ensure all the generated texts fit into the desired formats.
  3. Building a custom multi-agent framework means that we have to have robust logic for the coordination of each ML model. One of the challenges is with the speed of processing. Optimization efforts are made to support a more comfortable experience.
  4. We also spend a lot of time deciding and tweaking how the backend shall communicate with the server to achieve maximum efficiency.

Accomplishments that we're proud of

  1. We accomplished > 30 hours of sleep in total as a team.
  2. We have great pizza at Harvard Square
  3. We all survived a 12+ hour flight.
  4. We were able to hook up to and start using the Eleven Labs API in less than 30 minutes
  5. We achieved a really high level of refinement

What's next for Synapse

  • Cultural Awareness — Different cultures express and interpret emotions in unique ways. Synapse aims to adapt its emotion detection and visualization models across languages, dialects, and cultural norms, ensuring empathy translates globally.

  • Personalized Emotion Models — Fine-tuned per user based on their communication style, tone, and emotional baseline, so the system grows more accurate and intuitive over time.

  • Cross-Platform Integration — Embedding Synapse’s emotional intelligence into existing messaging platforms such as Discord, Slack, and WhatsApp.

Ultimately, our goal is to make communication more than informational —
to make it empathetic, personal, and culturally intelligent.

Built With

Share this project:

Updates