NovaCare – AI Healthcare Workflow Automation

AI agents that automate healthcare tasks using Amazon Nova: voice-powered patient intake, document understanding, and end‑to‑end UI workflow automation for clinics and hospitals. #AmazonNova

Comment 1

## Inspiration

The idea for NovaCare came from observing the long lines, manual paperwork, and overwhelmed staff in public hospitals and clinics. Elderly patients struggled to register, nurses typed data manually into outdated systems, and doctors relied heavily on handwritten prescriptions and paper records.
I realized that with advances in agentic AI, multimodal models, and voice technology, many of these repetitive workflows could be automated, freeing healthcare workers to focus on real patient care.
This inspired me to build NovaCare—an AI-powered automation system using Amazon Nova.

## What it does

NovaCare is an end‑to‑end healthcare workflow automation system powered by Amazon Nova.

It includes:

  • 🎤 Voice-based patient assistant (Nova 2 Sonic)
    Allows patients to register, book appointments, and ask questions through voice interaction.

  • 🧾 Medical document understanding (Multimodal Embeddings)
    Extracts medicines, diagnosis, observations, and dosage from handwritten prescriptions and lab reports.

  • 🧠 Reasoning & triage engine (Nova 2 Lite)
    Summarizes patient symptoms, recommends departments, and generates structured records.

  • 🤖 UI automation using Nova Act
    Automates real hospital/insurance portals—filling forms, uploading documents, booking appointments.

All interactions are shown in a modern dashboard, making healthcare much more efficient.

## How we built it

1. Architecture & Design

I started by designing a modular system that connects voice, documents, reasoning, and UI automation under one workflow.

Voice Input → Nova Sonic  
Documents → Nova Multimodal Embeddings  
Reasoning → Nova 2 Lite  
Automation → Nova Act Agents  
Frontend ↔ Backend ↔ Database

2. Frontend

  • Built using React, TailwindCSS, and Vite
  • Includes voice capture UI, document upload, agent activity viewer, and a patient dashboard

3. Backend

  • Developed using FastAPI (Python)
  • Handles Nova API calls, orchestrates agent tasks, stores data, and manages workflows
  • PostgreSQL for structured data
  • Amazon S3 for storing images/PDFs

4. Nova Integrations

  • Nova 2 Sonic for speech-to-intent
  • Nova Multimodal Embeddings to understand images + text in medical documents
  • Nova 2 Lite as the reasoning layer
  • Nova Act to automate step-by-step interactions on web portals

5. Agents

Created UI automation agents capable of:

  • Clicking buttons
  • Typing patient details
  • Uploading files
  • Navigating multi-page hospital portals
  • Handling failures and retries

## Challenges we ran into

1. Handwritten prescriptions

Extracting structured data from different handwriting styles required multiple prompting iterations and validation.

2. Making UI automation reliable

Web portals often:

  • Load slowly
  • Have inconsistent layouts
  • Use hidden elements
    This required robust agent workflows with fallback logic.

3. Real-time voice performance

Tuning buffer sizes, session contexts, and chunking logic was essential for smooth back-and-forth voice interaction.

4. Coordinating multiple Nova models

Ensuring that Sonic, Lite, Embeddings, and Act all worked together seamlessly required careful orchestration.

## Accomplishments that we're proud of

  • Built a fully functional, multimodal, voice-driven AI system
  • Created agents capable of navigating real-world healthcare portals
  • Successfully extracted structured data from handwritten prescriptions
  • Designed a clean UI that shows real-time agent actions
  • Delivered a project with genuine real-world impact potential, not just a theoretical demo

## What we learned

  • How to design agentic AI systems that combine planning, reasoning, and execution
  • Best practices in multimodal embedding extraction
  • How to build reliable UI automation flows using Nova Act
  • How to combine voice, text, and images into a unified AI workflow
  • Importance of error‑tolerant, human-friendly UX in healthcare solutions

## What's next for NovaCare – AI Healthcare Workflow Automation

  • Expanding voice support for more regional Indian languages
  • Adding doctor dashboards & analytics
  • Integrating diagnostic image analysis (X‑rays, scans)
  • Deploying pilots in clinics or NGOs for real-world testing
  • Adding telehealth features for remote consultations
  • Automating insurance reimbursement workflows end-to-end

Built With

  • 2
  • access
  • act
  • agent
  • agents
  • ai-&-amazon-nova-amazon-nova-2-lite-?-reasoning
  • amazon
  • amazon-web-services
  • and
  • api
  • apis
  • architecture
  • audio
  • automation
  • axios
  • backend
  • browser
  • capture
  • ci/cd
  • client-side
  • code
  • communication
  • containerization
  • css
  • data
  • database
  • definitions
  • deployment
  • devops
  • docker
  • document
  • documentation
  • ec2
  • embeddings
  • external
  • fastapi
  • files
  • for
  • framework
  • frontend
  • gateway
  • github
  • headless
  • iam
  • image
  • interactions
  • json
  • jwt
  • lambda
  • management
  • models
  • multimodal
  • nova
  • openapi
  • orm
  • pipelines
  • postgresql
  • prescriptions
  • primary
  • pydantic
  • python
  • react
  • relational
  • repository
  • routing
  • runtime)
  • s3
  • secure
  • sonic
  • speech
  • speech-to-speech
  • speech-to-text
  • sqlalchemy
  • storing
  • summarization
  • swagger
  • tailwind
  • traffic
  • triage
  • ui
  • understanding
  • uploaded
  • utilities
  • vite
  • web
  • workflow
  • workflows
  • yaml-based
Share this project:

Updates