Multilingual Symptom Triage Front Door

A WhatsApp-first multilingual AI agent that safely guides patients to self-care, clinic appointments, or urgent care while bridging language gaps between patients and healthcare providers.

Comment

Inspiration

Patients who don't speak the clinic's language often wait longer, repeat themselves, or never get through at all. Front-desk staff spend hours asking the same intake questions, and interpreters aren't always available. Emergency warning signs like chest pain or stroke symptoms can get buried in a routine queue when intake is rushed or misunderstood.

Patients already live on WhatsApp. We wanted to give clinics a front door there that speaks the patient's language, captures a clean intake, and never lets an AI make the emergency call.

What it does

A patient messages the clinic on WhatsApp in Hindi, Spanish, English or any language.

The system:

  • Checks every message for red flags first. A deterministic, clinician-reviewed rule engine looks for chest pain, stroke warning signs, severe breathing difficulty and loss of consciousness. It checks both the original language and the English translation, before any AI runs. A red flag at any point sends the approved emergency message and emails staff immediately.
  • Runs a structured intake. One question at a time: main symptom, duration, severity, age group and relevant history.
  • Routes to a safe next step using a fixed, clinician-approved workflow:
    • mild adult cases get approved self-care guidance in their language
    • other cases are offered the 3 soonest free appointment slots
    • anything unclear goes to a human
  • Books by text. The patient replies with a slot number and confirms. Retries and duplicate WhatsApp deliveries never double-book.
  • Hands off to the clinic. Staff get a structured English summary by email (original and translated text, extracted fields, routing outcome) and a dashboard of sessions, transcripts, safety events, appointments and metrics.

How we built it

  • Backend: Python, FastAPI and pydantic v2, with PostgreSQL (Alembic migrations) and an in-memory store for tests.
  • Channel: WhatsApp through AWS End User Messaging Social. Inbound events arrive via Amazon SNS to a webhook that verifies signatures and drops redelivered messages.
  • Language and clinical text:
    • Amazon Comprehend detects the language
    • Amazon Translate translates to English and back
    • Amazon Comprehend Medical extracts clinical entities
  • Safety gateway: a YAML rule set in English trigger phrases and a fuzzy matcher. It runs on every message.
  • Intake: a pure, enum-typed state machine checked by mypy --strict. No session state lives in process memory.
  • Generative AI: Amazon Bedrock (Nova Lite) in three narrow, guarded roles:
    • turning free-text answers into a value from a fixed allowed set
    • adding one short, warm acknowledgement line
    • drafting a staff handover note
  • Notifications: Amazon SES sends the staff summary emails.
  • Dashboard: read-only React, Vite and TypeScript.
  • Testability: every AWS service sits behind a port interface with an in-memory fake, so the full test suite runs without AWS credentials.

Challenges we ran into

  • Keeping AI useful without letting it touch clinical decisions. We had to design guardrails so Nova Lite could help without ever deciding escalation or severity.
  • Safety across three languages. Misspellings, mixed-language messages and negation ("no chest pain") can't be parsed safely by a keyword check. So the matcher deliberately treats any trigger phrase as a potential red flag. It returns "uncertain", never "clear", when translation fails.
  • Reliable booking on a messaging channel. WhatsApp can deliver the same message twice, and two patients can pick the same slot. We made booking idempotent with per-slot locking.

Accomplishments that we're proud of

  • A red-flag decision never depends on model output. The safety gateway runs before any Nova call, and the agent has no tool to bypass it.
  • Nova Lite output is tightly guarded:
    • a model-inferred "mild" is never accepted
    • acknowledgement lines are capped at 160 characters
    • medical-advice words are blocked in all three languages
  • If Bedrock is unavailable, everything falls back to the deterministic path.
  • Concurrent tests prove that many simultaneous bookers for one slot produce exactly one appointment.
  • Staff get a clear English summary that separates original text, translated text, missing fields and the routing outcome.

What we learned

  • Amazon Comprehend Medical only works on English text. So the order of the pipeline matters: Amazon Comprehend detects the language, Amazon Translate converts the message to English, and only then does Comprehend Medical extract clinical entities. Its confidence scores taught us to treat low-confidence entities as hints for staff, not confirmed patient facts.
  • AWS End User Messaging Social made WhatsApp a native AWS channel. We send replies through the socialmessaging API, and inbound messages arrive as Amazon SNS HTTPS notifications. so we learned to verify every SNS signature against its signing certificate before trusting the payload.
  • SNS can deliver the same message more than once. We de-duplicate on the WhatsApp message ID and made booking idempotent, so a redelivered "1" never books twice or sends a second reply.
  • Amazon Bedrock is easy to call and needs boundaries. We used the Converse API with Amazon Nova Lite through its US cross-region inference profile. Each Nova call runs inside an SNS webhook delivery, so we gave the client tight timeouts and limited retries. If Bedrock is slow or unavailable, every AI role returns nothing and the deterministic path takes over.
  • Smaller models work well for narrow jobs. Nova Lite was fast and cheap enough to interpret short free-text answers and draft a staff handover note. Strict output checks made it safe to use in a clinical flow.
  • Amazon SES needs setup before the demo. The sender must be a verified identity, and in the SES sandbox every recipient must be verified too. Requesting access early saved us on demo day.
  • Interfaces in front of every AWS service paid off. Each service sits behind a port with an in-memory fake, so we built and tested the whole flow before WhatsApp and SES access was approved. Each real AWS integration is switched on through environment settings.

What's next for Multilingual Symptom Triage Front Door

  • A supervised pilot with a partner clinic using clinician-approved rules and self-care content. We'll measure intake completion, time to intake, booking conversion, and staff preparation time.
  • Integration with clinic scheduling and EHR systems.
  • Amazon Bedrock AgentCore for tool orchestration, and a serverless deployment for multiple clinics.
  • Configurability to choose the clinician's language.

Built With

Share this project:

Updates

Submission history