CareCompanion

Inclusive automation platform that bridges the digital gap for the elderly and digitally illiterate. Enables users to schedule medical appointments, and order prescriptions.

Comment

Inspiration

Many elderly individuals living independently face significant friction when trying to use modern smartphones, navigate web portals, or order prescriptions. For example, registering for a doctor via BPJS Mobile JKN or checking social benefits requires multi-step authentication, reading small text, and filling out complex digital forms. This complexity often leads to missed appointments, delayed medications, and high dependence on family caregivers. We wanted to build a "Zero-UI" assistant where the elderly only need to send a simple WhatsApp voice message or a photo of their medicine to get things done, returning their dignity through independence.

What it does

CareCompanion acts as an agentic companion that listens, understands, and acts on behalf of the user:

  • Voice/Vision Intake: An elderly user sends a WhatsApp voice note (e.g., "Book a cardiologist appointment for this Friday morning") or photos a prescription/pill bottle.
  • Intelligent Parsing: The backend uses Gemini 1.5 Flash to transcribe audio, detect intent, extract parameters, and perform OCR on prescription bottles, returning structured JSON data.
  • Explainable Review (Human-in-the-Loop): If the AI detects ambiguity (e.g., "gilingan biasa" or an unclear appointment time), it flags the transaction on a React-based Caregiver Dashboard with an explainability log, allowing a family member or nurse to verify the details.
  • Robotic Process Automation: Cleaned data triggers a UiPath Unattended Robot that logs into medical/hospital portals, schedules the consultation, downloads the reservation slip, and sends it back to the user via WhatsApp.

How we built it

The platform is built using a highly decoupled agentic architecture:

  1. Frontend: A React.js dashboard using Vite and Vanilla CSS. It provides caregiver oversight, displaying task draft cards, status trackers, and AI confidence/explanation logs.
  2. Backend: FastAPI (Python) for processing WhatsApp webhook payloads from Twilio, orchestrating calls to Gemini, and saving task records to Supabase.
  3. Database: Supabase (PostgreSQL) for holding patient profiles, task logs, audit trails, and executing real-time triggers.
  4. AI Layer: Vertex AI (Gemini 1.5 Flash) utilizing Structured Outputs to ensure consistent JSON responses for classifications, transcribing, and OCR.
  5. RPA Layer: UiPath Integration Service triggers unattended robots built with UiPath Studio Web/Desktop to automate the Hospital Booking and E-Pharmacy portals.

Challenges we ran into

  • Authentication & CAPTCHAs: Modern medical portals (like BPJS Mobile JKN) have strict CAPTCHAs and SMS-based OTPs, making unattended RPA challenging. We solved this by designing a hybrid attended/unattended model and integrating UiPath Action Center to route OTP inputs to caregivers when necessary.
  • Colloquial Language Processing: Elderly users often use regional dialects, vague terms, or slang. We solved this by prompt engineering Gemini to translate local contexts and flag low-confidence values as ambiguous instead of guessing incorrectly.
  • Real-Time Webhook Routing: Handling asynchronous tasks (waiting for the robot to complete a web-scraping task) required establishing Webhook listeners in FastAPI to update Supabase and notify the user via Twilio.

Accomplishments that we're proud of

  • True Zero-UI Accessibility: Building a system where the user does not need to learn a new application or even know how to type—relying entirely on voice and photography.
  • GenAI + RPA Synergy: Successfully coupling the reasoning capabilities of Gemini with the operational execution of UiPath, showcasing how AI agents can perform real-world browser actions securely.
  • Explainability First: Implementing a validation system that highlights exactly what fields the AI was confused about, ensuring security and correctness before performing transactions.

What we learned

  • Accessibility is a Core Design Principle: Technology should adapt to the user, not the other way around. Zero-UI systems are highly effective for elderly care.
  • Error Boundaries in Agentic Systems: Pure GenAI cannot safely perform database transactions or make bookings alone without guardrails. Combining it with structured RPA workflows and human-in-the-loop validation ensures enterprise-grade reliability.

What's next for CareCompanion

  • Regional Dialect Fine-Tuning: Expanding Gemini prompts to natively support local languages and Indonesian regional dialects (Sunda, Jawa, etc.) more fluently.
  • IoT Smart Dispenser Integration: Linking the platform to physical smart pillboxes to automatically trigger refill orders when pill levels are low.
  • Expanding Automations: Integrating with pension portals, social insurance schemes, and transportation networks (like Gojek/Grab) to automatically arrange patient transport to and from hospitals.

Built With

Share this project:

Updates