NovaCare MediAssist

Your medical records, finally speaking your language.

Comment

Nova Mediassist

"Because no one should leave the doctor's office confused."

Amazon Nova AI Hackathon 2026 · Multimodal Understanding · Agentic AI · Voice AI Built with Amazon Nova Pro · Nova Lite · Nova 2 Sonic · AWS Lambda · API Gateway · S3

Inspiration

We all face incidents where doctors handover report after a routine checkup , when doctor mentions the results are a little concerning we just keep the report and never look it again

Not because we didn't care. Because we had no way to understand it.

That moment stayed with me.

According to the National Assessment of Adult Literacy, nearly 36% of U.S. adults have basic or below-basic health literacy — meaning they struggle to understand prescription labels, discharge instructions, or lab results. The problem compounds with age, language barriers, and the sheer complexity of modern medical documentation.

The gap between what doctors write and what patients understand is not a small one. It is:

As medical records grow more detailed and doctor appointment times shrink to an average of 18 minutes, patients are increasingly left to interpret their own health data — alone, confused, and sometimes making dangerous decisions as a result.

Nova Mediassist was born from that gap.

The timing aligned perfectly with the Amazon Nova AI Hackathon — a chance to build something real, impactful, and technically ambitious using Amazon's most capable multimodal AI models. We didn't want to build another chatbot demo. We wanted to build something that could genuinely change how a 68-year-old woman relates to her own health.

What We Learned

This project was a deep technical and human education across multiple dimensions.

1. Multimodal AI is genuinely transformative for healthcare

Amazon Nova Pro's ability to ingest a PDF or image and reason across text, tables, and visual structure simultaneously is not a incremental improvement — it is a qualitative leap. A traditional NLP pipeline would require:

With Nova Pro, the entire pipeline collapses to a single model call:

The reduction in latency, infrastructure complexity, and failure surface area was dramatic.

2. Agentic tool calling changes how AI solves problems

One of the most important lessons was learning when not to answer from the model's parametric memory, and instead route to a live tool. For drug interactions especially, static knowledge is dangerous — drug recalls, new contraindications, and updated FDA warnings happen continuously.

The agent's decision process follows a routing function:

This hybrid approach — live tools for factual safety-critical data, language model for reasoning and explanation — produced dramatically more reliable and trustworthy outputs than either approach alone.

3. Voice AI changes who can use the technology

Text-based AI assumes literacy. Nova 2 Sonic removes that assumption entirely. An elderly patient, a visually impaired user, or someone in a moment of medical stress who simply cannot type — all of them can access Nova Mediassist through natural speech.

The Word Error Rate (WER) of Nova 2 Sonic in our tests on medical terminology (notoriously difficult for speech recognition) was significantly better than our expectations:

Where S = substitutions, D = deletions, I = insertions, N = total words. Medical terms like "creatinine", "HbA1c", "thrombocytopenia" were consistently recognized correctly — a testament to Nova Sonic's training breadth.

4. Serverless architecture enforces good engineering discipline

Building on AWS Lambda forced every function to be stateless, single-purpose, and fast. You cannot hide architectural laziness behind a persistent server. Every Lambda had to:

  • Complete within a bounded timeout
  • Handle its own error surface
  • Be independently deployable and testable

This constraint, while initially frustrating, produced cleaner, more maintainable code.

5. Healthcare AI requires a higher standard of humility

Perhaps the most important lesson: knowing what the AI should not do. Nova Mediassist is designed to explain and inform — never to diagnose, never to prescribe, never to replace clinical judgment. Every response includes appropriate caveats. Every drug interaction check links back to FDA sources. The system is designed to increase a patient's ability to have an informed conversation with their doctor — not to replace that conversation.

How We Built It

Architecture Overview

Nova Mediassist is a fully serverless application built entirely on AWS with no persistent compute infrastructure.

┌─────────────────────────────────────────────────────────┐
│                    USER INTERFACE                        │
│         Static HTML/CSS/JS hosted on S3 + CloudFront    │
└──────────────────────┬──────────────────────────────────┘
                       │ HTTPS
┌──────────────────────▼──────────────────────────────────┐
│                  API GATEWAY (REST)                      │
│         /analyze    /agent    /voice                     │
└──────┬───────────────┬──────────────┬───────────────────┘
       │               │              │
┌──────▼──────┐ ┌──────▼──────┐ ┌────▼────────┐
│  Lambda     │ │  Lambda     │ │  Lambda     │
│  analyze   │ │  agent      │ │  voice      │
│  (Nova Pro) │ │  (Nova Pro  │ │  (Nova Pro  │
│             │ │  + Tools)   │ │  + Polly)   │
└──────┬──────┘ └──────┬──────┘ └────┬────────┘
       │               │              │
┌──────▼───────────────▼──────────────▼───────┐
│           AMAZON BEDROCK                     │
│   Nova Pro v1 · Nova Lite v1 · Nova Sonic   │
└──────────────────────────────────────────────┘

Component Breakdown

Frontend — S3 Static Website

A single index.html file with zero external dependencies beyond Google Fonts. Pure HTML5, CSS3, and vanilla JavaScript. Design philosophy: fast, accessible, works on any device. The file is served from S3 with CloudFront CDN for global low-latency delivery and HTTPS.

Key UI features:

  • Drag-and-drop document upload with base64 encoding
  • Real-time animated loading states during Nova processing
  • Voice recording via the Web Audio API (MediaRecorder)
  • Audio playback of Polly-synthesized responses
  • Fully responsive — works on mobile for bedside use

Lambda 1 — Document Analysis (/analyze)

Receives a base64-encoded PDF or image. Constructs a multimodal Nova Pro request with both the document and a structured JSON prompt. Parses the response into four clinical categories:

Output Field Description
summary Plain-English 2–3 sentence overview
abnormal_values List of out-of-range findings
medications Medications identified in document
urgent_flags Items requiring immediate medical attention

The prompt engineering for reliable JSON output took significant iteration. The final approach uses a strict schema declaration in the prompt with explicit instructions to return nothing outside the JSON envelope.

Lambda 2 — Agentic Chat (/agent)

Implements a multi-turn agentic loop using Nova Pro's native tool-calling capability. The agent has access to three tools:

Tool 1: check_drug_interactions Queries the OpenFDA REST API in real-time:

Tool 2: explain_medical_term Routes to Nova Lite (lower cost, faster) for simple explanations:

Tool 3: check_normal_ranges Uses Nova Lite to assess whether a lab value is within expected clinical ranges, with age and sex context where provided.

The agentic loop iterates until stopReason == "end_turn", with a maximum of 5 iterations to prevent infinite loops:

while iterations < MAX_ITER:
    response = nova_pro.invoke(messages, tools)

    if response.stop_reason == "end_turn":
        return final_answer

    if response.stop_reason == "tool_use":
        results = execute_tools(response.tool_calls)
        messages.append(tool_results)
        iterations += 1

Token efficiency was carefully managed. The total token budget per agent conversation is bounded by:

Where T_{max} = 4096 tokens was set to balance response quality against cost.

Lambda 3 — Voice AI (/voice)

Three-stage pipeline:

  1. Transcription — Nova Pro processes the WAV audio and returns the transcript
  2. Response generation — Nova Pro generates a concise voice-optimized reply (2–4 sentences, no markdown, natural speech cadence)
  3. Speech synthesis — Amazon Polly Neural TTS (Joanna voice) converts text back to MP3, returned as base64 for browser playback

The voice pipeline is designed to complete in under 8 seconds end-to-end, which is the threshold for acceptable conversational latency:

Technology Stack

Layer Technology Purpose
Frontend HTML5 · CSS3 · Vanilla JS Zero-dependency UI
CDN AWS CloudFront HTTPS + global delivery
Storage AWS S3 Static file hosting
API AWS API Gateway (REST) Endpoint routing
Compute AWS Lambda (Python 3.12) Serverless functions
AI — Vision/Text Amazon Nova Pro Document analysis + agent
AI — Fast/Cheap Amazon Nova Lite Term explanations
AI — Voice Amazon Nova 2 Sonic Voice understanding
Speech Amazon Polly Neural Text-to-speech responses
Drug Data OpenFDA API Real-time drug information
IaC AWS CLI + Bash Deployment automation

Challenges We Faced

Challenge 1 — Reliable Structured Output from a Multimodal Model

Getting Nova Pro to consistently return valid JSON from a document analysis — especially for edge cases like handwritten prescriptions, poor-quality scans, or multi-page PDFs — was harder than expected.

Early attempts produced responses like:

Here is my analysis of the document:

Summary: The patient has...

Instead of clean JSON. The solution was a combination of:

  1. Explicit schema declaration in the prompt
  2. A safe_parse_json() function that strips markdown fences and falls back gracefully
  3. Low temperature (T = 0.1) to reduce creative deviation from the format

The reliability function improved from approximately 72% clean JSON on the first iteration to ~97% after prompt engineering:

Challenge 2 — CORS in a Serverless Architecture

Cross-Origin Resource Sharing (CORS) between an S3-hosted frontend and API Gateway is deceptively complex. Each Lambda needs to return correct headers on both POST and OPTIONS (preflight) requests. Missing a single header on a single method causes silent failures in the browser with no useful error message.

The solution: every Lambda handler checks httpMethod == "OPTIONS" first and returns a complete CORS envelope before any other logic executes.

Challenge 3 — Managing Token Costs in the Agentic Loop

The agentic loop can balloon in cost if not carefully managed. In early testing, a single complex question involving multiple tool calls consumed over 8,000 tokens — roughly $0.03 per query, which would become significant at scale.

Optimization strategy:

Route sub-tasks to Nova Lite where possible:

Cap conversation history to the last N turns:

The result: average cost per agent interaction dropped from $0.028 to $0.006.

Challenge 4 — Voice Latency on Mobile Networks

The round-trip voice pipeline (record → encode → upload → transcribe → generate → synthesize → download → play) introduced latency that felt uncomfortable on slower mobile connections.

Key optimizations:

  • Compress audio before upload (target < 100KB for a 10-second clip)
  • Stream Polly audio output rather than waiting for full synthesis
  • Show intermediate "I heard you say..." transcript immediately while generation runs

This brought perceived latency from ~12 seconds to under 6 seconds on a typical 4G connection.

Challenge 5 — Medical Accuracy and Responsible AI

The hardest challenge was not technical. It was ethical.

When a real patient uploads a real lab report and asks "is this dangerous?" — the stakes are high. Getting this wrong in either direction is harmful: false reassurance could delay needed care; unnecessary alarm could cause panic and distrust.

Our approach:

  • Always recommend consulting a doctor for medical decisions
  • Cite source (FDA, clinical reference) for drug information
  • Use hedged language: "this value is above the typical reference range" rather than "you have kidney disease"
  • Never generate a diagnosis — only explain findings
  • Flag anything marked urgent in the original document

This philosophy is baked into every system prompt across all three Lambda functions.

Impact & Vision

Nova Mediassist addresses a problem that affects billions of people globally — not just in the United States. The WHO estimates that poor health literacy costs the U.S. healthcare system alone over $230 billion annually in preventable complications, hospital readmissions, and medication errors.

The communities who benefit most are also the most underserved:

  • Elderly patients managing multiple chronic conditions
  • Non-native speakers navigating a foreign healthcare system
  • Rural communities with limited access to specialist follow-up
  • Patients with disabilities who benefit from voice-first interaction

By building on Amazon Nova's multimodal and voice capabilities, Nova Mediassist can serve all of these groups from a single, accessible, serverless interface that costs less than $5 to run during an entire hackathon period.

The vision beyond this hackathon:

  1. Multilingual support — Nova's multilingual capabilities could serve patients in their native language
  2. EHR integration — Direct connection to Epic / Cerner patient portals via FHIR API
  3. Medication reminder agent — Proactive Nova Act-powered reminders based on prescription schedules
  4. Specialist triage — Automatically suggest which specialist to consult based on findings

Acknowledgements

Built with using Amazon Nova on AWS Bedrock during the Amazon Nova AI Hackathon 2026.

Special thanks to the AWS team for making frontier multimodal AI accessible to every developer — and to every patient who has ever sat in a waiting room holding a piece of paper they couldn't understand.

This project is for them.

#AmazonNova · #AWSBedrock · #HealthTech · #ServerlessAI · #Nova Mediassist

© 2026 Nova Mediassist · Built for Amazon Nova AI Hackathon · Submitted March 2026

Built With

Share this project:

Updates

Submission history