BuckyUp

Inspiration

Throughout the renowned Captain America series, Winter Soldier Bucky Barnes served as a faithful companion. Unfortunately, Bucky also suffered from serious PTSD. This project was inspired by Bucky’s story—I hoped to build something that can help prevent mental health crises like that of Bucky and almost 23% of American veterans through efficient, human-less triage (1). As of this writing, the US lacks a unified mental health screening system for deployed soldiers and the social stigma surrounding mental illness inhibits many from seeking help (2).

As a premed student, I’ve seen firsthand that all decision making in healthcare is holistic. With this in mind, BuckyUp integrates numbers, sights, and sounds from the user.

What it does

Mental health triage needs to be faster, smarter, and more effective—especially in high-stakes environments like the military, where mental health is inextricably tied to performance. BuckyUp is an AI-driven solution that performs rapid psychological assessments in just five minutes. Pulling together real-time conversational AI (GPT-4o), computer vision (MediaPipe), and biometric data (Terra API), BuckyUp evaluates speech patterns during a classic SIGECAPS-based screening interview and measures blink rate and vitals to flag at-risk individuals dynamically. Why measure eye blinks and heart rate? Both have been shown to increase in clinically depressed patients (3, 4).

By overlaying these insights into a visual report, we empower healthcare professionals to make informed, data-backed decisions. BuckyUp also offers a guided meditation for interested users—empowering them to prioritize mental health.

How we built it

I leveraged ElevenLabs for voice generation, enabling a calming and immersive guided meditation experience. My real-time conversational AI agent was built on GPT4o, enhancing interactivity by picking up on user feedback through verbal cues. We also collected biometric data via the webcam, which was running my custom computer vision model built with MediaPipe. The tech stack featured a Next.js frontend for a seamless user experience and a Flask backend for efficient processing and integration.

Challenges we ran into

One of the biggest challenges I faced was tackling this project as a solo developer and first-time hacker! However, I soon realized that being on a solo team didn’t mean I had to work alone—I could collaborate and co-work with other TreeHacks attendees. From mastering the integration of Terra’s APIs to build the perfect tool, to fueling up on enough caffeine to power through the night, the challenges were relentless. Yet, the support of my newfound friends made the journey all the more rewarding.

I also found it tough to integrate the various components of the algorithm together on the web app, particularly streaming the Terra API data while also running the Computer Vision model.

Accomplishments that we're proud of

Built an end-to-end app by myself for the first time. Having spoken to doctors at Stanford Med, it’s clear that AI is a real application for triage—the need is there, and it’s exciting to have developed an application which is truly practical and relevant for healthcare professionals.

What we learned

Rather than building algorithms from scratch, it’s often easier to start from baseline, open-source models. User experience is a crucial element of app success. Caffeine stops working after a while.

What's next for BuckyUp

The current app is largely a proof of concept to which a host of additional features can be applied—from syncing additional Terra API data to developing a more comprehensive interactive assessment. After conducting additional needs-assessments among healthcare experts, I hope to continue building out BuckyUp’s features and test in military users to assess the precision of the risk assessment score.

References

(1) “Va.Gov: Veterans Affairs.” How Common Is PTSD in Veterans?, 24 July 2018, www.ptsd.va.gov/understand/common/common_veterans.asp#:~:text=One%20study%20found%20that%20among,in%20Veterans%20using%20VA%20care. (2) Vermetten, Eric et al. “Deployment-related mental health support: comparative analysis of NATO and allied ISAF partners.” European journal of psychotraumatology vol. 5 10.3402/ejpt.v5.23732. 14 Aug. 2014, doi:10.3402/ejpt.v5.23732 (3) Mackintosh, J H et al. “Blink rate in psychiatric illness.” The British journal of psychiatry : the journal of mental science vol. 143 (1983): 55-7. doi:10.1192/bjp.143.1.55 (4) Moser, M et al. “Increased heart rate in depressed subjects in spite of unchanged autonomic balance?.” Journal of affective disorders vol. 48,2-3 (1998): 115-24. doi:10.1016/s0165-0327(97)00164-x

Built With

• elevenlabs
• flask
• mediapipe
• next
• openai
• supabase
• terra