Empathy

Welcome to Empathy!
GIF
Recording
Moment Capturing
Your Moment

Inspiration

We spent a lot of time choosing an idea that we can all really relate to before starting the project, and came up with 10,000+ options late into the first night of the hackathon. Through rounds of discussions on what we really wanted out of this hackathon, we realized the most amazing part of events like Cal Hacks is the people that it brings together. We wanted a better way to connect to all the cool people we've only had the time to meet through long lines for boba and free t-shirts. Thus, we decided to create Empathy— something that could capture collective memories at events and inspire new connections through this shared experience.

What it does

We interface you with a curated and empathetic AI Voice Agent that asks you about your day, the event you're at, and what else you're up to. We then generate a custom "artifact" that represents your current state— a playful collection of hand-drawn assets specifically arranged and designed for the story you told. We summarize your day as a small keepsake in this way, and also match you with other people who might be the same in the most esoteric of ways.

How we built it

We utilize VAPI's voice agent to interface with the user and React/NextJS to build the frontend / site. We host a custom fuzzy auditory search model via OpenAI's Whisper API on a nginx reverse proxy Flask server running on an AWS EC2 instance, and we utilize Supabase to store users and their stories. We finally use Gemini for its long context capabilities to search for coincidences that extend past simply word2vec cosine similarity and embrace the human complexities of natural language.

Challenges we ran into

We ran into several issues, both technical and creative.

A big part of Empathy, as you can tell, is how we interface with the AI that helps power the experience. Members of our team had fundamentally different beliefs on how we should itnerface with AI and that led to many long discussions— we were ultimately helped out by some judges who offered their own takes on what an ideal experience would look like.

From a creative perspective, we needed to create personal artifacts that can be representative of potentially vastly different experiences in a way that could inspire more conversation. We didn't think a pure AI generated image can have the impact we hope to create, so we ultimately composed 150+ original sketches that our app will choose intelligently from based on the interview and compose in a more cohesive collage.

From a technical perspective, this was our first time hosting our own server and coordinating data through a shared database through it and a website as well. We spent a lot of my time dealing with encodings of auditory formats, searching through non-textual media, etc. Many moving parts also meant that sequences could be unreliable, and we felt those problems only exploded once we moved to deploying on Vercel. Ultimately, we were able to overcome most of our challenges but ultimately did not incorporate audio search into our product as it was too unreliable.

Accomplishments that we're proud of

I am personally proud of our team for boldly presenting and debating every idea we had, in order to ultimately find one that we were all interested in— this was my first time working with some of them and I'm glad it worked out the way it did. We also challenged all of ourselves technically (most complex project for those of us second-timers), and explored a domain we might not have individually. I'm glad we built this with a goal in mind and a focus on community rather than ourselves, in a way that has influenced the projects I now think about building for the future.

What we learned

We learned how to completely and vividly express our visions for a product and come together to align n them, how to deploy and the challenges of production, how quickly some of us got used to speaking to an AI agent, and how comfortable carpet can be on 2 hours of sleep. We also learned how to use powerful frameworks like NextJS and unfortunately struggled through things like CORS before learning about the API layer of Next (a lesson to be well remembered, no doubt)

What's next for Empathy

We obviously have many features we'd like to ship next, from audio to more powerful search and more beautiful visualizations. But first, we want to get as many people using it as possible to see what its truly capable of!