HealthKaki

Inspiration

Healthcare support in Singapore is extensive, but understanding it can still be difficult, especially for elderly patients and their caregivers. Medical bills, discharge summaries, subsidy breakdowns, and medication labels are often filled with unfamiliar terms, codes, and instructions. For seniors who are less comfortable with English or digital systems, this can create unnecessary confusion and anxiety.

Research from Duke-NUS CARE found that limited English health literacy was a significant driver of medication non-adherence among elderly Singaporeans, specifically because many could not understand English prescription medication labels, leading to uncertainty about whether they were taking their medications correctly (Tan et al., 2020). The language barrier between a senior and the label on their medicine bottle is not a minor inconvenience, it is a patient safety risk.

We were inspired to build HealthKaki as a simple healthcare companion that helps seniors better understand the documents they receive after visiting a clinic or hospital. Our goal was to make important healthcare information easier to access with just one action: taking a photo, and then having it explained back to you, in your language, in large text, and read aloud.

What it does

HealthKaki is a mobile-friendly web application. Users sign in securely with their NRIC and date of birth and complete a short one-time profile (citizenship status, household income, and household size), which lets the app tailor results to their situation.

From there, they can upload or take a photo of:

Medical bills and receipts
Referral letters and discharge/specialist summaries
Prescription slips / Printed medication labels

The application extracts the information from the document and presents it in a clearer, senior-friendly format.

For medical bills, HealthKaki:

Breaks down complicated charges
Explains subsidies already shown on the bill
Highlights potentially relevant support schemes, personalised using the user's profile
Presents the estimated out-of-pocket cost clearly

For medication labels, HealthKaki:

Extracts the printed medication name and instructions
Explains the common purpose of the medication in simple language
Translates the printed instructions
Reads the explanation aloud for users who may struggle with written English
Refuses to interpret handwritten labels, a built-in safety check rejects any label that isn't officially printed, because a misread dose could be dangerous

Everything is designed for seniors and caregivers:

Four languages, English, 中文, Bahasa Melayu, and தமிழ், for both the interface and the scanned content
Read-aloud on every result
Accessibility built in, adjustable text size, a high-contrast mode, and large touch targets throughout

The application only explains instructions already printed on official labels. It does not generate new dosage advice or replace professional medical guidance.

How we built it

We built HealthKaki as a mobile web application, developing it spec-first with Kiro, writing structured requirements, a design document, and a task breakdown (including formal correctness properties for privacy and safety) before implementing each feature. Our stack includes:

Next.js and React for the frontend and application structure
Tailwind CSS for a clean, responsive, and senior-friendly interface
Supabase for authentication, the subsidy database, and document storage
Google Gemini (multimodal AI) for reading documents and extracting structured information, and for translating the scanned content on the fly
DeepL (backed by a Singapore-healthcare glossary) for translating the fixed interface text
A hybrid text-to-speech system, the browser's built-in voices for English and Chinese, and Google Cloud Text-to-Speech for Malay and Tamil, so explanations can be read aloud in every supported language

The user begins by selecting the type of document they want to scan. After uploading an image, Gemini extracts the relevant text and the system organises it into clear sections. For bills, the extracted details are compared against a database of Singapore healthcare support schemes, such as CHAS, Pioneer Generation, Merdeka Generation, MediSave, MediShield Life, and MediFund, and filtered against the user's profile (age, citizenship, and income per person) so the schemes shown are genuinely relevant to them. For medication labels, the system focuses on the official printed instructions, such as dosage frequency and whether the medicine should be taken before or after food. The results are displayed using large text, high contrast, simple wording, and clear visual cards to improve accessibility for elderly users.

Because medical documents contain sensitive personal data, any NRIC numbers are automatically detected and redacted before results are shown or stored, and uploaded images are processed in server memory rather than being kept.

Challenges we ran into

One of the biggest challenges was dealing with the complexity of Singapore's healthcare subsidy system. Subsidy eligibility may depend on several factors that are not always visible on a medical bill, such as household income, age, scheme membership, treatment type, and healthcare provider. We addressed part of this by collecting a short user profile (age, citizenship, and household income) to personalise which schemes are surfaced, but we were careful not to let the system claim to guarantee exact eligibility from a single uploaded document. Instead, HealthKaki separates information into categories such as:

Subsidies already applied
Schemes that may be relevant
Additional information required
Official channels for verification

Another major challenge was making read-aloud actually work in all four languages. Translating the text turned out to be only half the problem: we found that most devices simply don't have Malay or Tamil voices installed. Malay would be read aloud with an English accent, and Tamil produced complete silence, because an English voice cannot pronounce Tamil script at all. We iterated through several approaches before landing on a hybrid design that uses free on-device voices where they exist and a cloud voice where they don't, with caching and a clear on-screen notice when audio isn't available.

We also had to ensure accuracy when reading medical bills and medication labels. Poor image quality, unusual layouts, abbreviations, and OCR errors can all affect the extracted information, so we added retries and fallback parsing, and deliberately restricted the medication scanner to official printed labels, actively rejecting handwritten ones rather than risk a dangerous misread.

Privacy was another key concern. For the hackathon prototype we used synthetic documents, implemented automatic fail-closed NRIC redaction (if redaction ever fails, the request is rejected rather than risk exposing an identity number), and avoided permanently storing uploaded files. A production version would add stronger privacy and security controls on top of this foundation.

Accomplishments that we're proud of

We are proud that we turned a complex healthcare problem into a simple and accessible user experience. Within a short hackathon period, we designed a working application that lets users scan medical documents / medication prescriptions to receive clear explanations of bills, subsidy information, and printed medication instructions.

We are especially proud of:

Designing the application around the needs of elderly users by including accessibility features such as adjustable text sizes, high-contrast mode, and multilingual support across the interface, scanned results, and read-aloud audio in English, Chinese, Malay, or Tamil.
Presenting complex information using simple language, adjustable text, and clear visual cards
Combining document scanning, AI-based explanation, translation, and text-to-speech into one experience
Personalising subsidy relevance using a lightweight user profile instead of guessing from the bill alone
Building privacy and patient safety in from the start, fail-closed NRIC redaction and a fail-safe handwriting check
Recognising the risks of inaccurate medical or subsidy information and designing the system to avoid overclaiming
Building a solution that addresses a real and meaningful problem in Singapore

Most importantly, we are proud that HealthKaki does more than process documents. It helps users feel more confident when understanding important healthcare information.

What we learned

Through this project, we learned that solving a real-world problem requires more than simply applying AI to a document.

The information must also be:

Accurate
Easy to understand
Accessible to the target user
Presented responsibly
Clear about its limitations

We learned that "supporting four languages" and having them actually work on a real device are very different things, especially for text-to-speech, which depends entirely on what voices a user's device happens to have. We learned that fail-closed and fail-safe defaults are the right call when the stakes are someone's privacy or their medication. And we learned that designing spec-first with Kiro kept a fast-moving, feature-rich build coherent, writing the requirements and safety properties before the code meant privacy and safety were designed in, not bolted on.

Above all, we learned the importance of designing for elderly users. A technically powerful system may still be difficult to use if the interface contains small text, too many steps, or complicated language. This encouraged us to focus on simplicity, accessibility, and trust instead of adding unnecessary features.

What's next for HealthKaki

In the future, HealthKaki could be expanded with:

Additional spoken languages and dialects (e.g Teochew, Cantonese etc.)
More reliable, wider-coverage read-aloud audio across all languages
Direct links to official subsidy application and verification portals
Caregiver accounts for helping elderly family members
Reminders based on printed medication schedules
Support for more healthcare document formats
Fully on-device redaction of sensitive information
Upgrading our current NRIC login to integrated Singpass authentication
Integration with official healthcare systems, subject to approval and security requirements

Our long-term vision is for HealthKaki to become a trusted digital companion that helps seniors and caregivers better understand healthcare costs, subsidies, and medication instructions.

How to try HealthKaki

1. Open the app Go to https://healthkaki.vercel.app/

2. Create an account (Login screen) Enter:

NRIC/FIN: any valid NRIC. (M + 7 digits + any letter, e.g. M1234567A, M7654321Z - each unique number creates its own fresh account.)
Date of birth: any date. To see age-based schemes like Pioneer Generation, use an older date such as 01/01/1948.

Press Continue. A new account is created instantly (no email or password needed).

3. Complete the one-time profile (Onboarding screen) This is what personalizes your subsidy results. For the fullest demo, enter:

Full name: (optional)
Citizenship status: Singapore Citizen
Year you became a citizen: (optional)
Household monthly income: choose a lower range (e.g. the lowest option) to qualify for income-based schemes like CHAS
Household size: e.g. 2

Press Continue.

4. Use the app You'll land on the Home screen. From here you can:

Scan a medical bill / referral / prescription → see a subsidy breakdown
Scan a medication label → get a plain-language explanation
Switch language (English / 中文 / Melayu / தமிழ்) and press Read Aloud on any result
Adjust text size and high-contrast mode in Settings