eHealth

Inspiration

Telehealth services are expected to grow at an impressive rate of 39% year over year until 2030, a growth, in part, driven by AI assistants, chatbots, and modern distribution technologies. At the same time, we're facing a growing elderly care crisis. The global population of people aged 65 and older is set to reach 1.5 billion by 2050 – double what it is today. This dramatic demographic shift coincides with a severe healthcare staffing shortage. In the United States alone, we're expecting a shortfall of 86,000 physicians by 2036. This shortage is especially concerning in elderly care facilities, which are already struggling with low nurse-to-patient ratios. However, we believe there's hope. By strategically implementing AI technology in areas and ways where it's needed most, we can create lasting positive change that touches lives across the world. Though we had only 36 hours, we wondered if we could build an autonomous robotic assistant which harnessed the accessibility of telehealth, the overwhelming need for elderly care, and the efficiency of AI-powered nursing support? eHealth was born, a robot which we hope can inspire the world to create tools which will stand for the benefit of everyone.

What it does

eHealth is a multimodal agentic system which has the autonomy to move itself simply given a request to pick up and drop off anything in its environment. It is powered by two Visual Language Models, or VLMs, hosted locally on an Nvidia Jetson, and uses a Raspberry Pi to make API calls to both secondary devices and cloud computed LLMs. Beyond its tech, eHealth was made to be as helpful as possible to as many people as possible. In the context of disability support, it autonomously identifies and retrieves objects in the environment based on natural language requests or app-based instructions. The agent's ability to be controlled via a joystick provides an additional layer of accessibility for users with different abilities, or for remote home and family safety. In the field of healthcare, eHealth can help with basic patient care tasks, freeing up healthcare workers for more complex duties or more emotionally impactful encounters. The remote monitoring capability allows healthcare providers to oversee multiple rooms efficiently while maintaining patient privacy and safety.

In mental health settings, the agent serves as both a physical assistant and a consistent presence, helping patients maintain routines and complete daily tasks while providing data about activity patterns through its monitoring system.

How we built it

At the beginning of TreeHacks, our team scrounged up spare parts and within a few hours had built out a basic chassis to hold our robot. Having attended an Nvidia talk where they demoed their impressive Jetson Nano micro AI computers as well as NanoOwl, a live object detection VLM, we realized that this was a perfect opportunity to create what would become eHealth. We positioned webcams to act as the eyes of a dual VLM system that gave our robot a complete understanding of its environment so it could be as helpful as possible. We wanted eHealth to be a light in the dark, a helpful friend to anyone who wants to use it. By combining both VLMs to contextualize its environment through multi-layered natural language processing and calculating the position of objects in 3 dimensional space utilizing hitbox depth software, we apply multiple layers of LLM inference to allow eHealth the ability to parse user inputs as a task or query, going through different decision trees for each of the two inputs. The task utilizes image recognition software string parsing from Gemini, feeds the data from this parsing into the VLM (NanoOwl) to do hitbox detection on “important” objects, wherein the 3d displacement process occurs with both movement and claw tracking.

Challenges we ran into

The first challenge that we ran into was optimizing the latency between the web interface and the physical robot actions as the web interface ran gen-ai image and video inference based off of robotic actions. From more of a UI/UX design perspective, creating an intuitive user interface for different user groups so our product could be accessible in the healthtech sector was of utmost importance. A smaller but pivotal challenge was making sense of documentation that consumed tens of hours from our team, as we had to implement robust security for remote control, ensuring reliable object recognition across various lighting conditions, and balancing autonomous operation with human oversight added to the complexity. Finally, the cost of development with such a hardware and software intensive project was exorbitant as we had to navigate the challenge of integrating siloed AI backend systems with the frontend project while effectively utilizing Jetson Nano and Raspberry Pi as both computational and physical detection and action objects.

Accomplishments that we're proud of

After countless hours of troubleshooting, debugging, and refining, our team finally built the ecosystem that is eHealth successfully, developing a multi-modal control system that adapted to different user needs, which ensured that the robot had a variety of use cases that would help assist the healthcare sector along with elderly care for years to come. A team favorite integration was a secure and responsive web interface allowing for smooth remote monitoring, while real-time video streaming minimized latency, making interactions feel natural. We were incredibly excited with such a system as it took over 30 hours of straight coding and hardware development for everyone on the team. The cherry on top was this level of inherent reliable communication between the web application and the physical robot tied everything together. At 5 AM, exhausted but exhilarated, we celebrated our success in the only way that made sense—a victorious 10-minute nap before diving back into the next challenge.

What we learned

This project was more than just a technical challenge: it was a deep dive into the complexities of designing assistive technology that truly works for people. We learned firsthand how user-centered design shapes interactions, with UI/UX playing a crucial role in making systems intuitive and accessible. One of our team's biggest learnings stemmed from optimizing real-time communication in robotics, which pushed us to explore AI backend integrations. As a whole, eHealth really allowed us to understand the power of large language models and image detection systems. Balancing automation with human control also came with ethical considerations, particularly in ensuring AI doesn’t interfere with high-risk prompts while also recognizing its potential impact on livelihoods. Making complex systems accessible to the diverse user groups who are involved in receiving healthcare required creative problem-solving, as did integrating multiple technologies into a single, cohesive platform. Each challenge taught us something new, reinforcing the importance of designing with both technology and people in mind.

What's next for eHealth

The future of our Robotic Agent is filled with possibilities, with a strong focus on transforming healthcare support. We plan to expand its integration with existing telehealth platforms, enabling seamless remote care. More sophisticated healthcare monitoring capabilities will enhance real-time assessments, while specialized routines and behaviors tailored for elderly care will provide personalized assistance. A network of connected devices will allow for large-scale institutional deployment, ensuring consistent support across hospitals, nursing homes, and assisted living facilities. AI-powered predictive care capabilities will further strengthen its ability to anticipate patient needs and improve outcomes. eHealth’s vision is to create a scalable solution that not only eases the healthcare staffing crisis but also enhances patient care while preserving the essential human connection through remote monitoring. Designed for flexibility, eHealth’s ecosystem will continue evolving with user needs and technological advancements, ensuring long-term sustainability in the ever-changing healthcare landscape.