GIF
Demo
Landing Page
Randomizing Avatar
Default AI avatar
Response to a prompt from user
Randomizing on mobile app
Mobile app

HealMate

Introduction

HealMate is an AI-powered virtual therapist designed to provide personalized mental health support and resources to users seeking to improve their well-being. By combining interactive features with user customization, HealMate aims to create a safe and engaging environment for mental health assistance. We plan to enhance the experience further by incorporating audio responses, allowing the virtual therapist to communicate verbally with users.

Features

Virtual AI Therapist: Engages users in empathetic conversations, offering coping strategies, emotional support, and therapeutic guidance. The virtual therapist can respond in various ways to cater to individual needs, with plans to add audio responses for a more immersive experience.
Mood Journaling: Allows users to log and track their emotions over time, helping identify patterns and triggers.
Resource Library: Offers articles, breathing exercises, relaxation techniques, and other resources to manage stress, anxiety, and other mental health concerns.
Emergency Contacts: Quick access to helplines and counseling services for urgent support.
Customizable Therapist Avatar: Users can personalize the virtual therapist's appearance using interactive 3D models and custom assets, making interactions more engaging and relatable.
Adobe Creative Assets: Integrated custom stickers and icons from Adobe Explore Add-ons to enhance user experience, aiding in presentations and data visualization, especially useful for marketing and e-commerce roles.

Inspiration

The rising mental health challenges faced by individuals worldwide inspired us to create HealMate. We recognized that many people hesitate to seek help due to stigma, accessibility issues, or lack of personalized resources. Our goal was to develop a platform that provides empathetic support and encourages proactive mental health management. The idea of a virtual AI therapist emerged from the need to simulate a real therapeutic experience, offering users the comfort and confidentiality they might not find elsewhere. By integrating AI technology with interactive features and custom visuals, we aimed to create a tool that feels personal, engaging, and supportive.

How We Built It

Frontend

React and Vite: For building a fast and responsive user interface.
Three.js: Utilized for rendering 3D graphics, allowing users to customize the virtual therapist avatar.
@react-three/fiber and GLTF Transform: For seamless integration of 3D models and real-time rendering.
TailwindCSS: Employed for rapid and responsive styling.
Leva: Used to create intuitive control panels for customization options.
Zustand: Implemented for efficient state management across components.
Adobe Explore Add-ons: Integrated custom assets like stickers and icons to enhance user engagement and visual appeal.

Backend

FastAPI and Uvicorn: Chosen for building a high-performance API to handle requests efficiently.
LangChain and LangGraph: Integrated for advanced AI capabilities, enabling complex conversational experiences with therapeutic context.
Pydantic and pyhumps: Used for data validation and serialization, ensuring data consistency.
WebSockets: Implemented for real-time communication between the client and server.
External Knowledge Sources: Leveraged APIs like Wikipedia and Tavily Python to enrich the virtual therapist's responses.

Deployment and DevOps

Terraform: Employed for infrastructure as code, automating the deployment process to Vercel.
GitHub Actions: Set up for continuous integration and deployment (CI/CD), automating testing and deployment workflows.
Docker and DockerHub: Containerized both frontend and backend applications for consistent deployment across environments and published images to DockerHub.

Tools

GLTF Transform: Used for optimizing 3D models, ensuring smooth performance in the browser.
PocketBase: Implemented as a lightweight backend solution for data storage and real-time updates.

Challenges We Faced

Complex Integration: Merging AI capabilities with interactive 3D customization required bridging multiple technologies and ensuring they worked seamlessly together.
Therapeutic Response Generation: Crafting AI responses that are empathetic and therapeutically appropriate was challenging, requiring careful tuning and testing.
Performance Optimization: Rendering complex 3D models without compromising application performance was a significant challenge.
Deployment Automation: Setting up Terraform with GitHub Actions for seamless deployment to Vercel involved a steep learning curve but was crucial for efficient development.
Containerization: Ensuring our Docker images were optimized and worked consistently across different environments posed several challenges.
Audio Integration Planning: Designing the system to accommodate future audio features added complexity to our architecture.

Accomplishments We're Proud Of

Seamless Integration: Successfully created a smooth integration between the AI virtual therapist and the interactive frontend components.
Therapeutic Engagement: Developed an AI that can provide empathetic and therapeutically appropriate responses, enhancing user support.
Automated Deployment: Implemented a robust CI/CD pipeline using Terraform and GitHub Actions, streamlining our development and deployment process.
User Engagement: Enhanced user experience through customizable avatars and the integration of Adobe Creative Assets.
Containerization: Successfully dockerized the application, facilitating consistent and efficient deployment.
Performance Improvements: Optimized 3D rendering to ensure smooth performance across devices.

What We Learned

Therapeutic AI Development: Gained insights into creating AI models that can deliver therapeutic conversations responsibly.
Technology Integration: Gained valuable experience in integrating AI technologies with interactive web applications.
DevOps Practices: Learned how to automate deployment processes using Terraform and GitHub Actions.
Containerization Skills: Improved our ability to create and manage Docker containers for application deployment.
3D Graphics Optimization: Enhanced our understanding of rendering 3D graphics efficiently in a web environment.
Asset Management: Learned how to create and integrate custom assets using Adobe Explore Add-ons.

Future Enhancements

Audio Responses: Implement audio capabilities for the virtual therapist, allowing it to respond verbally for a more immersive experience.
Advanced Personalization: Implement machine learning algorithms to adapt responses based on user interactions.
Mobile Applications: Develop native mobile apps for iOS and Android to increase accessibility.
Community Features: Introduce forums or group sessions to foster community support among users.
Localization: Add support for multiple languages to cater to a global audience.
Wearable Integration: Incorporate data from wearable devices to provide more personalized support.
Enhanced Analytics: Implement analytics dashboards for users to track their mood patterns and progress over time.

Built With

Frontend

React, Vite, Three.js, @react-three/fiber, GLTF Transform
TailwindCSS, Leva
Zustand
Adobe Explore Add-ons

Backend

FastAPI, Uvicorn, Pydantic, pyhumps, python-multipart, WebSockets
LangChain, LangGraph
Wikipedia API, Tavily Python

Deployment & DevOps

Terraform, GitHub Actions, Docker, DockerHub
Vercel

Database

PocketBase

Tools

GLTF Transform

Acknowledgments

We thank the hackathon organizers for providing this opportunity and our peers for their support and feedback throughout the development process.

Additional Information

HealMate is more than just a project; it's our commitment to making mental health support accessible and engaging. We believe technology can play a pivotal role in breaking down barriers and providing personalized assistance to those in need. By evolving the virtual AI therapist to include audio responses, we aim to simulate a more lifelike therapeutic experience, helping users feel heard and supported. Through HealMate, we hope to contribute positively to the well-being of individuals and communities.

Thank you for considering our submission!

Built With

adobe-express-add-ons
docker
dockerhub
fastapi
github-action
gltf
langchain
langgraph
leva
pocketbase
pydantic
react
react-three/fiber
tailwindcss
terraform
three.js
uvicorn
vite
websockets
zustand

Submitted to

HackNJIT 2024

Created by

As the developer responsible for the visual identity and user engagement tools on HealMate, I focused on creating a logo that reflected the supportive, friendly nature of our mental health platform. The design aimed to be approachable and empathetic, conveying a sense of trust and warmth that aligns with our mission.

In addition, I integrated Adobe Explore Add-ons to provide users with custom stickers and icons, enhancing interactivity and making the user experience more engaging and personalized. These assets played a crucial role in the frontend, helping to create an inviting atmosphere and aiding in visual storytelling, especially with the 3D avatar customization feature.

I also documented the project thoroughly, ensuring that both the technical setup and user-facing features were clearly explained. This included step-by-step installation instructions, a tech stack overview, and usage details, which helped make the project accessible and streamlined for new developers and users alike. Balancing visual design, user engagement, and comprehensive documentation allowed me to contribute to a cohesive and user-friendly product experience on HealMate.

Mohamad Alrajjal
As a frontend engineer on the HealMate project, I worked on building a highly interactive experience using React, Vite, Three.js, and Zustand for smooth state management and 3D graphics. Deploying this complex app brought its own set of challenges, especially as we set up automated infrastructure with Terraform, Docker, and Vercel.

One significant hurdle was integrating Terraform with Vercel. Since Vercel doesn’t natively support Terraform, we had to work around this by using GitHub Actions to manage environment settings and ensure smooth CI/CD. Dockerization was essential for ensuring consistent builds, but configuring it to handle Vite’s dev server and large 3D assets required careful tuning. Managing secrets securely across environments and coordinating deployments with GitHub Actions added complexity, especially with quick iteration needs.

Performance optimization was also crucial. We used GLTF Transform to reduce 3D model sizes and set up a CDN for faster asset loading. Overall, balancing infrastructure demands with a high-quality frontend experience was a valuable learning process, especially in managing deployment and performance for an engaging, responsive app.

Shaik Tabrez
As the backend engineer on HealMate, I focused on creating a reliable API using FastAPI and Uvicorn to support the frontend and enable AI-driven virtual therapy. Integrating LangChain and external resources like the Wikipedia API was essential for delivering intelligent, empathetic responses. I used Pydantic for data validation to ensure clean, structured data, which was especially important for maintaining accuracy in AI responses and mood tracking.

Our backend was designed to handle complex interactions and provide a seamless experience to users. Optimizing for performance and scalability, I ensured that the architecture was adaptable for future features like audio capabilities without requiring significant restructuring. Building a secure, high-performance backend that could support real-time, AI-based interactions was a fulfilling challenge and allowed me to deepen my expertise in creating resilient, user-centered applications.

jgwentworth92