Inspiration
According to the American Medical Association (AMA), physicians complete an average of 45 prior authorizations per week, consuming nearly two full business days. This administrative burden delays critical patient care and is a leading cause of burnout among healthcare professionals. Current RPA (Robotic Process Automation) solutions fall short because they rely on brittle, hardcoded scripts that break whenever an insurance portal updates its UI. I realized that to truly solve this, I needed an agent that could see and understand a web page just like a human does. This inspired me to build NovaCare, an autonomous clinical web agent powered by Amazon Nova 2.
What it does
NovaCare is a multimodal, autonomous Web Agent designed to navigate complex medical insurance portals (like UnitedBlue, Medicare, etc.) to submit prior authorizations, check claim statuses, and file appeals.
- Multimodal Extraction: The user uploads a patient's medical PDF, and the application extracts exhaustive clinical data into a structured JSON payload using Nova 2 Lite.
- Autonomous Navigation: Through a natural language command, the agent opens the portal and completes the mission autonomously.
- Human-in-the-Loop: If the agent encounters a missing mandatory field or a complex roadblock, it halts its process and asks for help via a specialized "Orb" interface. Once the user replies, the agent instantly resumes its task.
How I built it
I architected NovaCare around the See-Think-Act cognitive paradigm, heavily leveraging the AWS Bedrock ecosystem:
- The Brain (Think): I used Amazon Nova 2 Lite as the core reasoning engine. I provided it with a robust System Prompt defining operational rules, safety guardrails, and a suite of JSON-schema tools.
- The Eyes & Hands (See & Act): I built a custom engine using Playwright. Instead of passing raw HTML to the LLM, I developed a JavaScript payload that scans the DOM and returns a clean, filtered list of interactive elements. Nova 2 Lite evaluates this state and triggers tools like
click_elementortype_text. - The Interface: I developed an asynchronous Streamlit frontend featuring a dynamic, CSS-animated "Orb" that reflects the agent's internal state (Listening, Thinking, Acting, Waiting).
Challenges I ran into
- Zero-Hardcoding Navigation: Building an agent that does not rely on pre-defined CSS selectors was a major technical hurdle. I had to fine-tune the DOM-parsing script so that Nova 2 Lite could accurately map logical actions to dynamic web elements without getting confused by hidden or disabled UI components.
- Preventing AI Hallucinations in Healthcare: In the medical field, an AI cannot guess clinical data. I solved this by creating a strict
ask_user_for_helptool. Orchestrating the pause of a synchronous Playwright backend while pinging the asynchronous Streamlit frontend required building a custom Inter-Process Communication (IPC) loop using temporary file handshakes. - Managing State in Streamlit: Keeping the UI responsive and the CSS animations synced while a heavy web-automation process ran in the background required careful thread management and session state architecture.
Accomplishments that I'm proud of
- True Autonomy: I successfully built an agent that can dynamically adapt to UI changes. It does not just click blindly; it reads the screen, verifies if its previous action succeeded, and plans its next move.
- The Human-AI Collaboration: The seamless transition from autonomous execution to the "Human-in-the-Loop" fallback is my proudest achievement. It proves that AI agents can be safe, reliable, and collaborative in high-stakes environments.
What I learned
I gained deep expertise in the AWS Bedrock API, specifically in utilizing the Tool Use capabilities of the Amazon Nova model family. I learned how to write highly constrained System Prompts to keep an autonomous agent on track. I also learned advanced Playwright techniques for DOM evaluation and interaction.
What's next for NovaCare
My next step is to migrate the underlying orchestration to the Amazon Nova Act service to manage fleets of agents more reliably. I also plan to deeply integrate Nova 2 Sonic to enable full real-time speech-to-speech conversations during the Human-in-the-Loop interruptions, allowing doctors to guide the agent entirely hands-free.
Built With
- amazon-bedrock
- amazon-nova
- amazon-polly
- boto3
- css
- html
- javascript
- playwright
- pypdf
- python
- streamlit