jARvis

AR HUD assistant designed for surgeons that transforms users’ heart rate variability into a live, visual display of stress level - if dangerous, jARvis triggers a warning and mindfulness intervention.

Comment 1

Inspiration

Medical providers are under enormous daily stress. Research has shown that excess stress leads to a poor learning environment for trainees and is associated with worse surgical performance, increased risk of serious procedure-related complications, and a higher rate of intraoperative mistakes.

What it does

jARvis is a smart, real-time stress monitoring assistant designed for an augmented reality HUD. jARvis transforms a continuous stream of user biometric sensor data into a live, visual representation of emotional state made more accurate over time via machine learning. If the user’s stress level reaches an evidence-based, dangerous threshold associated with critical error, jARvis will alert its user and recommend a brief mindfulness breathing intervention to help down-regulate emotional state.

How we built it

jARvis was built in Arduino and Unity.

Challenges we ran into

Connecting our hardware biometric sensor to Unity via Bluetooth.

Accomplishments that we're proud of

Connecting our hardware biometric sensor to Unity via Bluetooth. Making an immersive Unity experience from almost zero Unity experience.

What we learned

Working with hardware is HARD! From designing a wearable device, to ensuring accurate data collection, to testing proper device functionality, to linking hardware with software, the “hardware” track challenged us and helped us to divide our diverse skills in order to triage problems in order of priority.

What's next for jARvis

jARvis will require rigorous study to become an evidence-based medical tool. If validated, jARvis may be developed for other industries that require high-stakes, high-stress decision-making, such as in military aviation, Formula 1 racing, and NASA space-walks, where remote monitoring via real-time data streams are needed for the most accurate and immersive feedback experiences.

Built With

Share this project:

Updates

posted an update —

here are my notes from the night after day 1:

hi team! we made a lot of progress today on the bricks we need for tomorrow. as a review, we:

  1. evaluated and rejected a proprietary hardware oximetry platform in favor of an open source one that is faster, cheaper, and lower power. yay open source! go china! @tanpesto led the way through this effort, and hand-assembled the board we cannot use! @mikejsz went to the store and purchased it out of pocket, thank you!
  2. evaluated and rejected a gut-wrenchingly bad AR headset in favor of the facebook-produced one that is more open to developers. yay…. facebook? ew zuck. @Hanqing Tang (Ariela) (She/Her) somehow is immune to the evil headset. never forget this power.
  3. started the unity scene that will be the setting of the experience that our judges will get. i can’t believe how much clicking i did in unity, following the vive docs, only for it to never work correctly with the headset. wtf, HTC?
  4. implemented some designs on how we can provide pleasant feedback reflecting the user’s emotional state. we have code, we have geometry, we are in a good place! @prasanth stepped in at the last minute of our teambuilding last night, and i’m so glad he did.
  5. collecting and modeling the 3d assets we need to create an immersive environment. we had to learn the annoying way that there are not many freely available assets for this application area, and some freely available assets are not freely usable in open source projects! @Hanqing Tang (Ariela) (She/Her) did a ton of research and modeling for this today.

i’m not even sure what else i missed, because i was focused on my rectangle and didn’t get to see a lot of the other collaborations! i was so enthused to review the contents of the table before leaving and see indexed cards that i never witnessed the creation of, full of encoded thoughts.

Log in or sign up for Devpost to join the conversation.