mysight

A long term PTSD/Anxiety recovery toolkit built on Spectacles

Comment

Videos

mysight

mysight leverages augmented reality through Snap Spectacles to create a supportive, real-world mental health aid for individuals with PTSD and/or anxiety. Rather than removing users from reality, mysight enhances their surroundings with therapeutic AR tools that promote grounding, emotional regulation, and gradual exposure. The goal is to empower users to recover progressively and safely, while maintaining dignity, privacy, and control over their healing journey.

Inspiration

When we heard the topic of dreams, we had to spend a lot of time pondering. We thought on the real world concept; What is a dream? How are dreams different from nightmares? We thought on the metaphorical; What do dreams symbolize? How do they reflect our hopes and goals? Is to dream, to be free to pursue them? We examined our own backstories and personal experiences, and came across a solid conclusion. Everyone should have the freedom to dream. However, too often in our lives does mental illness take us away from that. Stressors from life build anxiety, fear, panic in our hearts regardless of past present or future. We are often constrained by these invisible shackles, making it difficult to fully achieve our true goals. We thought of veterans in our lives whose fear has built up over time, and made them unable to live a normal life. This tool is for them, this tool is for us, and this tool is for you.

We want to remove that sense of helplessness, and release peoples true potential through a realistic progression that meets users where they are already at.

What it does

mysight is a multifaceted toolkit, including the censoring of triggers, grounding strategies for panic attacks, and a long term audio journal powered by Snap Cloud. To block triggers it runs an on device YOLO image recognition model, and then places black boxes over all triggers the user has selected. Grounding strategies come from our team's research into PTSD literature combined with a calming bubble popping activity. The audio journal serves as a way to record your thoughts - prompting you to process the emotions healthily, and at your own pace. This feature is most helpful when used in conjunction with a licensed therapist, and other external coping methods - allowing for a seamless integration into traditional therapy, whilst helping to continually drive improvement.

How we built it

mysight was built from the ground up for the Snap Spectacles. Using Snap Cloud we where able to orchestrate audio recording and playback for the journal logs. Using Blender we where able to animate bubbles popping to create an appealing aesthetic and calming action. In Lens Studio we modified the rendering pipeline to swap the Snapchat Yellow for calming purple hues on certain UI elements. We also trained a custom gun and dog detection model on top of YOLO v7 nano, using a data set of over eight-thousand images.

Challenges we ran into

We ran into challenges with unfamiliarity and lack of easily available resources for Lens Studio. As with all newer platforms, there is less material available until the community is able to interact with it. However, this challenge was often reduced by the helpfulness of the Snap Developers readily assisting us at the event. Particular thanks go to Alessio and Jesse, without them, this project would not have been possible.

Further challenges existed with the unpredictable weather that occurred. An incoming snow storm cut our development time by many hours, shifting windows, and requiring fundamentally faster programming, video editing, and research. Luckily, thanks to amazing team members, and an iron-clad will to avoid scope creep, we finished within the nick of time. This isn't to say that we don't have future plans - the potential of these glasses is well known to us, and we want to add more, polish, and make this into a piece of gold; however, we cannot defy the power of time.

Accomplishments that we're proud of

This competition pushed us to go further than we have before. Being a 60+ hours of coding hackathon, we we're able to make a product that could go into use on this very day. Some of our greatest highlights are

  • Having found medical research to back our efforts
  • Training a working AI model locally, on a teammate's 5060
  • Creating custom 3D graphics using blender - pre rendering to increase performance
  • Reworking the rendering system in Lens Studio for custom colored integrated graphics with clean intuitive controls

What we learned

We learned over the course of this project the value of teamwork, responsible AI usage, and reaching out to the invaluable mentors attending the hack. During this project, many times when teammates got stuck, simply having another person to reach out to, ask questions, and explain our thinking with enabled us to overcome the bugs. AI was another thing the team had limited experience with using, but under the encouragement of Jesse and Alessio from Snap we tried out using Cursor, and found the ability to process a large codebase in a small period of time to be very useful, pointing out files related to objectives we had in our coding. Last, but certainly not least, was reaching out to people with experience. It is not possible to summarize the joy we had in working with many amazing mentors and developers. There is so much yet to be done in XR, and we are so excited to be a part of it.

What's next for mysight

  • Increasing the list of available triggers to censor
  • Adding a widened variety of coping mechanisms
  • Optimizing the model to run more smoothly in real time on the Spectacles
  • Therapy-ready report generation with AI insights for data driven analysis

Built With

  • blender
  • snap
  • snapcloud
  • snapml
  • spectacles
  • supabase
  • torch
Share this project:

Updates