UrTherapist

API testing using postman, got the API and endpoints from the Azure AI services , then I tested it .
LLM testing with the emotions recognized from face
API testing.
Azure AI services used.
Main webpage for a particular portal , for a person. His previous sessions will be loaded,
As you can see , it gives empathetic responses , caring and friendly.

Inspiration::

Honestly, I needed a therapist 6 months ago. I was in a guilty ride for the mistakes I made in college! I needed some good person to assist me on the good path and empathize with me in my realization and guiltiness. I realize I need to lock myself in studies and improve myself for a better future for me and my mother. I got the idea from a movie called Her (Joaquin Phoenix) and the Lucifer series.

What it does:

UrTherapist is a multi-modal, AI-powered mental health assistant that acts like a virtual therapist. It uses facial expressions (via webcam), tone of voice (via mic), and user messages to detect your emotional state in real time. If you’re feeling down, it offers a warm, empathetic response and practical coping techniques. If you’re happy, it celebrates that moment with you. If you're anxious or overwhelmed, it supports you like a kind friend — no judgment, just empathy. All interactions are stored securely, and users can view past sessions, mood stats, and even receive personalized therapy suggestions based on their emotional history.

How we built it

Frontend: React (Vite), TailwindCSS, Framer Motion Backend: FastAPI (Python), REST APIs AI Services: Azure Face API—to detect emotion from facial expressions Azure Speech-to-Text—to transcribe user voice Azure Text Analytics—to analyze sentiment from voice text Azure SQL Database—to store session data and emotional insights Azure Cognitive Search—to fetch therapy recommendations based on user mood Together.ai (Mistral 7B) --as the core empathetic therapist chatbot Authentication: Firebase Login/Signup with email verification Postman: Used to test all Azure API integrations

Challenges we ran into

Emotion detection from facial landmarks required custom logic on top of Azure Face API’s raw data. Speech-to-text sometimes struggled with accent/pronunciation, but we handled it with retry logic. Syncing multiple data sources (face, voice, text) into a unified emotional state took some experimentation. GitHub Copilot was great — but sometimes overconfident 😄. We had to guide it smartly, especially while integrating Azure APIs. Database server connection and the Ip address made issues , I used my college wifi, the blocked some of the port numbers.. took time to figure out ..

Accomplishments that we're proud of:

Built a fully functional multi-modal mental health app in just a few days Successfully integrated six Azure services in a real-time, user-facing workflow Created a deeply empathetic AI therapist using open-source LLMs Got the camera, mic, text, and backend to all talk together seamlessly Showcased real user emotion adaptation in UI and responses I created this whole project as a single person , which I'm really proud of myself for. Actually, my final exams are this week; I came to know about this hackathon lately. I couldn't miss this opportunity to take part. I just wanted to use this opportunity to prove my worthiness in this world. I'm confident I'll win this hackathon.