Pegasus

Catch burnout before it catches you.

Comment 1

Inspiration

Often times, when school assignments bog us down, we find ourselves stressed without even knowing it. The stress interferes with our daily life, and all the while, we prefer to just not think about it and not go out of our way to care about mental health. When presented with the opportunity to help individuals work through stress, we realized that the best way to solve this problem was not to create some incredibly elaborate setup that forces you to decide whether or not you’re feeling stressed, but to create an app so simple and inconspicuous that using it isn’t a distant idea for an already stressed person. Looking at technological innovations like Oura Rings, we sought to make Pegasus, which incorporates mental health checks subtly and calmly.

What it does

Pegasus is a simple app that works like a check engine light for your mental health. Instead of making you stop and decide whether or not you're stressed, it quietly pays attention to the small things that usually give stress away, like the way your typing slows down, how you react to a quick image we text you, and your facial expression and tone of voice when you get on a short call with it. It takes all of that and turns it into one calm wellness score, where a higher number means you're doing better, with one small suggestion to go along with it. You can also just talk to the app and it talks back to you out loud, and it will text you a gentle check-in during the day. It can even show you which parts of your brain are reacting the most using the Tribe v2 model. The whole point is that caring about your mental health shouldn't feel like a big effort, kind of like how an Aura ring keeps track of your sleep without you ever really thinking about it.

How we built it

We built Pegasus as a mobile app in React Native and Expo that talks to a few small Python services in the background. For the language and emotion side we used Hugging Face, with one model that reads facial expressions, another that reads sentiment, and a chat companion that we grounded with RAG over a small mental health dataset so its responses actually stay relevant. That companion uses sentence transformer embeddings to find the right information and a Qwen model to write the reply. For the brain side we used Meta's Tribe v2, which predicts how the brain reacts to a stimulus, and we ran it on Modal's GPUs so we could compare your reaction against a healthy baseline. We gave the app a voice using NVIDIA Riva, with Magpie for the text to speech and Parakeet for the speech to text, so it can listen and speak during a call. We used Bloo.io as our CPaaS to send the iMessage check-ins. To get the final number, we wrote a scorer that takes all the signals, including what you wrote, your typing, your face, your voice, and the Tribe reading, and combines them into the one score you see, and we used SQLite to keep the results. We also split our GitHub into a separate branch for each person so the four of us could work at the same time without overwriting each other.

Challenges we ran into

Almost every part of this gave us trouble at some point. Getting the Tribe v2 model to run at all, and then turning its brain reaction data into something our app could use, took a lot of trial and error before we finally got it working on Modal. Our scoring kept fooling us too, because someone could reply with something pretty dark and still end up with a score that said they were fine, so we had to teach it to notice when people were masking how they felt, like when someone says they're fine, really, don't worry, since that's exactly the kind of thing that makes burnout so easy to miss. We also ended up flipping the whole score so that a higher number means you're doing better. Finding a CPaaS that could actually send an iMessage was harder than we thought, and when our free trial ran out in the middle of building, we had to set up a backup so messages still reached the phone. Running the camera and the microphone and a back and forth voice conversation on a phone also caused some crashes we had to chase down, and getting four people's separate services to work together across different branches without breaking anything was something we had to stay careful about the entire time.

Accomplishments that we're proud of

We successfully created an fmri finetuned AI model that we ran on cloud servers and trained with Stanford's dataset

What we learned

This project required us to learn multiple different valuable technical skills, including finding viable CPaaS’s and keeping our GitHub repository separated into different branches so code wasn’t interefed with. Additionally, we learned valuable aspects of the brain’s reaction to stimuli when trying to utilize the Tribe v2 model, and just how much burnout and stress permeates our society even today.

What's next for Pegasus

Having completed a base version of our app, our goal from here on out is going to be to expand our app to our community and suggest it to others. We hope to get outreach at the personal level, not just the systemic one, because for mental health, a single person who is able to effectively deal with stress because of our app is a success in and of itself.

Built With

Share this project:

Updates