CapyGuardian

CapyGuardian: A wearable health system using an agentic AI to analyze health data, video & kinematics. It can detect falls and other abnormalities, instantly alerting others for assistance.

Comment

Inspiration

Imagine a world where care is both personalized and immediate—where every moment counts, and every individual’s unique needs are met instantly. CapyGuardian harnesses the power of agentic AI to blend wearable sensors with real-time health analysis delivering instant, tailored health insights. This approach transforms reactive care into proactive, individualized support that acts precisely when needed. Whether it's alerting a caregiver at the first sign of trouble or ensuring that a soldier in a high-stakes environment is safe, CapyGuardian is set to redefine emergency response. This dynamic system is a game-changer for older adults living independently, providing them with critical peace of mind, and for military platoons, where split-second decisions can make all the difference.

What It Does

CapyGuardian is a wearable health monitoring system that continuously tracks vital metrics such as heart rate, user kinematics, location, and even provides a waist-level video feed. All this data is presented on an intuitive dashboard where an integrated agentic AI analyzes the information in real-time. For example, if the AI abnormal heart rate patterns, it promptly issues an alert for immediate intervention—vital for both the vulnerable and large, dynamic groups.

The dashboard has 4 primary sections: 1) The feed from the camera at waist-level 2) The kinematics of the user displayed on a skeleton (the skeleton moves with the user) 3) Heart rate data 4) Location on a map with friendlies (green dots) and hostiles (red dots) displayed

Example Output from the CapyGuardian:

####################### I've submitted an urgent alert through the logs system. Based on my analysis:

  1. Critical Concerns:
  2. Person appears to have fallen (ground-level camera perspective)
  3. Heartbeat signal shows AFib indicators (uneven peaks)

  4. Person is alone in their location (map shows no nearby green dots)

  5. Environmental Assessment:

  6. Location: Office setting

  7. Lighting: Adequate for visibility

  8. Surroundings: Office furniture visible

  9. No immediate environmental hazards visible

  10. Medical Status:

  11. Possible cardiac event (AFib)

  12. Potential fall-related injuries

  13. Unconscious or limited movement (based on camera angle)

  14. Location Safety:

  15. Person (green dot) is in a building

  16. Several other people (red dots) are in nearby areas

  17. Location appears to be a monitored facility

IMMEDIATE ACTION IS REQUIRED. I've triggered the alert system and recommended immediate medical response team dispatch. The combination of a possible fall and AFib indication makes this a high-priority medical emergency. #######################

How We Built It

I built Capy Guardian using an Nvidia Jetson Nano 2GB Developer Kit for robust edge computing. Data is collected from 6 Adafruit IMU sensors (providing 3D acceleration and gyroscopic data) and a USB camera for a continuous chest-level video feed. Heart rate information is sourced via the Terra API, while real-time kinematics are processed from the IMU data using inverse kinematics with OpenSim. A Flask-hosted dashboard on the Jetson displays all these data streams, which are then analyzed by our agentic AI (via Scrapybara). Due to challenges in acquiring a static IP, to allow the agentic AI to interact with the dashboard, I upload prerecorded video feed and kinematic data to the agentic AI's ubuntu environment and host a demo dashboard there for it to analyze.

The real time kinematics was inspired by this paper link where it was implemented on a raspberry pi but here wanted to see if the upgrade to a jetson would be helpful.

Challenges We Ran Into

Compiling the OpenSim package on the Jetson Nano was particularly taxing—taking 4-5 hours and crashing multiple times, leading to initial instability in real-time kinematics due to package conflicts. Additionally, integrating multiple data streams while ensuring performance posed quite a challenge.

Accomplishments That We're Proud Of

I was actually super excited to see that Capy Guardian to detect critical health events, such as falls, from just some camera data and kinematic data! The integration of agentic AI to interact with and act upon real-time health data is extremely promising!

What We Learned

This project deepened my understanding of the challenges of compiling and optimizing large software packages on edge computing hardware. I also learned the importance of designing systems that are both user-centric and robust, ensuring accurate data collection and processing in real-time.

What's Next for Capy Guardian

Moving forward, I plan to integrate real-time streaming of location and heart rate data directly from devices like the Apple Watch, eliminating the need for the dummy data from Terra API. Also, a static IP for the Jetson Nano would allow us to test the agentic AI’s capabilities to handle more complex health scenarios. The IMUs instead of being taped onto my clothing, would be embedded into the clothes. Beyond elder care, there is significant potential for applications in sports and military environments, where continuous and timely health monitoring would be helpful!

Built With

Share this project:

Updates