Baymax

InspirationWe were inspired by the Global Caregiver Crisis. Hospitals worldwide are facing critical staffing shortages, with nurses experiencing record levels of burnout. The current "one-to-many" ratio means a single nurse often has to attend to dozens of patients simultaneously, leading to delayed care for basic needs.We asked ourselves: What if we could give every nurse an extra pair of hands?Inspired by the character Baymax, we wanted to build a Tele-Kinetic Nursing Assistant—a system that allows a single healthcare professional to remotely care for multiple patients, performing physical tasks without entering the room. This reduces infection risks, saves Personal Protective Equipment (PPE), and allows nurses to focus on high-level medical care rather than fetching water or adjusting trays.What it doesBaymax is a secure, voice-activated robotic assistant designed for isolation wards and elderly care facilities.Listens & Understands: The patient simply speaks ("Baymax, I need my water"). The system uses Speech-to-Text and Google Gemini 1.5 Flash to understand not just the keywords, but the intent and urgency of the request.Responds Naturally: Using Fish Audio, the robot replies with a comforting, human-like voice to reassure the patient ("I see your water cup. I am grabbing it for you now.")Sees & Acts: Using OpenCV and multimodal vision, Baymax scans the room, identifies the target object, and calculates the safe path to retrieve it.Remote Execution: The robotic arm (uArm Swift Pro) executes the movement smoothly. Crucially, this can be monitored or overridden by a nurse stationed miles away via our secure dashboard.How we built itWe built Baymax using a distributed Client-Server IoT Architecture:The Brain (Host): A high-performance machine running the cognitive stack. It handles MediaPipe for face tracking, OpenCV for object detection, and Google Gemini for reasoning.The Muscle (Edge): A Raspberry Pi connected to a uArm Swift Pro. It runs a lightweight Python server that translates high-level intents into G-Code motor commands.The Nervous System (Networking): This is our "secret sauce." We implemented a WireGuard VPN tunnel combined with custom NAT Traversal. This allows us to send low-latency UDP control packets securely over the public internet without exposing the robot to hackers.The Math: We didn't just use libraries; we wrote Custom Inverse Kinematics (IK) solvers to map 2D camera pixels into 3D servo angles $(x, y, z)$, ensuring the robot moves naturally rather than mechanically.Challenges we ran intoThe "Lag" Problem: Initially, chaining Vision → AI → Audio → Movement caused a 5-second delay. It felt robotic, not helpful. We solved this by building an Async Event Loop that triggers voice feedback and physical movement simultaneously (Parallel Processing), dropping latency to near-real-time.Secure Remote Access: Connecting to a Raspberry Pi inside a hospital network (behind strict firewalls) from the outside is usually a nightmare. Setting up the WireGuard peer-to-peer tunnel to punch through NAT without opening dangerous ports was our toughest infrastructure challenge.Hand-Eye Coordination: Calibrating the webcam so that "Pixel (200, 200)" equates exactly to "Reach 15cm Forward" required implementing a robust P-Controller (Proportional Controller) to smooth out the jittery tracking data.Accomplishments that we're proud ofDefense-Grade Security: We successfully secured an IoT device using modern cryptography (Curve25519 via WireGuard), ensuring patient privacy and preventing "Man-in-the-Middle" attacks.The "Handover" Feature: We achieved a specific autonomous sequence where the robot identifies a face, picks up an object, and extends it toward the person—mimicking a human handover.Multimodal Fluency: Creating a seamless loop where the AI sees, hears, speaks, and moves all at once.What we learnedRobotics is mostly Math: We learned that hardware is useless without good Inverse Kinematics. Writing the coordinate mapping logic gave us a deep appreciation for bio-mechanics.Latency is the Enemy of Empathy: For a robot to feel "caring," it must respond instantly. Optimizing our network stack was just as important as the AI prompts.Security cannot be an afterthought: Building the VPN architecture taught us that IoT devices are vulnerable by default and require a "Zero-Trust" networking approach.What's next for BaymaxMobility: Mounting the arm on a rover base to allow it to move between rooms.Soft Robotics: Replacing the rigid gripper with soft, pneumatic fingers to handle delicate medical equipment or fragile items.EHR Integration: Connecting Baymax to Electronic Health Records so it can remind patients to take specific medications at the right time