Inspiration
Healthcare organizations process thousands of medical documents every day, and reviewing those documents for coding and audit purposes is often repetitive, time-consuming, and highly dependent on human expertise. As an AI Engineer working on healthcare AI solutions, I wanted to explore how agentic AI could assist medical coding workflows while keeping humans in control of final decisions.
Rather than building another chatbot that simply answers questions, I wanted to build an intelligent agent capable of reading documents, retrieving relevant coding information, reasoning about evidence, calculating confidence, and automatically routing uncertain cases for human review. MedAgent-Gemini was created to demonstrate how AI agents can help streamline medical audit workflows in a safe and transparent way.
What it does
MedAgent-Gemini is an agentic medical coding and audit assistant that processes medical chart PDFs and performs a multi-step workflow.
The system:
- Extracts text from uploaded PDFs.
- Detects important medical sections such as HPI, ROS, MDM, Impression, Orders, Procedures, and Treatment Notes.
- Normalizes and analyzes clinical content.
- Retrieves CPT, HCPCS, and ICD candidates using local retrieval techniques.
- Uses Gemini to validate and reason over retrieved candidates.
- Calculates confidence scores for decisions.
- Routes low-confidence cases for human review.
- Creates GitLab review tasks when additional validation is required.
- Generates structured audit outputs.
The goal is not to replace medical coders but to assist them by reducing repetitive manual work and highlighting areas that require attention.
How we built it
We built MedAgent-Gemini using an agentic architecture that combines document processing, retrieval systems, LLM reasoning, and workflow automation.
Key technologies include:
- Python
- FastAPI
- Streamlit
- Google Gemini
- PyMuPDF
- pdfplumber
- BM25 Retrieval
- Docker
- Render
- GitLab Integration
The workflow begins with PDF extraction and section detection. Relevant evidence is then retrieved and analyzed using local retrieval mechanisms. Gemini acts as the reasoning layer, comparing retrieved candidates against the document evidence and producing a final assessment. Confidence scoring determines whether the result can proceed automatically or should be escalated for human review.
Challenges we ran into
One of the biggest challenges was balancing public demonstration requirements with healthcare data privacy requirements. Real medical coding databases and clinical documents contain confidential information and cannot be publicly shared.
To solve this, we redesigned the system to support synthetic demonstration datasets while preserving the full workflow architecture.
Another challenge was coordinating multiple AI components including retrieval, reasoning, confidence scoring, document processing, and workflow automation into a single coherent agent. Ensuring that these components worked together reliably required significant experimentation and testing.
Deployment and packaging for public evaluation while maintaining security and privacy was also a major engineering challenge.
Accomplishments that we're proud of
We are proud that MedAgent-Gemini goes beyond traditional chatbot functionality and demonstrates a true agentic workflow.
Key accomplishments include:
- Successfully deployed a live cloud-hosted version.
- Built an end-to-end PDF processing pipeline.
- Integrated retrieval, reasoning, validation, and workflow automation into a single system.
- Implemented confidence-based human review routing.
- Added GitLab workflow integration for review escalation.
- Created a privacy-safe demonstration environment using synthetic datasets.
- Delivered a complete prototype that can be tested publicly without exposing sensitive information.
What we learned
Through this project we learned that effective AI agents require much more than language generation.
We learned how retrieval systems, reasoning models, confidence scoring, and workflow automation can work together to solve real-world problems. We also gained valuable experience in designing systems that balance automation with human oversight, which is especially important in healthcare-related use cases.
Most importantly, we learned that agents become significantly more valuable when they can take actions, trigger workflows, and assist users in completing tasks rather than simply providing answers.
What's next for MedAgent-Gemini
Future development plans include:
- Advanced ICD-10 and CPT coding workflows.
- Support for larger medical knowledge bases.
- Enhanced reasoning and audit capabilities.
- Human-in-the-loop review dashboards.
- Multi-document case analysis.
- MCP-based enterprise workflow integrations.
- Secure cloud-hosted vector retrieval systems.
- Integration with additional healthcare and compliance workflows.
Our long-term vision is to create an AI-powered assistant that helps healthcare organizations improve efficiency, consistency, and audit readiness while maintaining strong human oversight and privacy controls.
Log in or sign up for Devpost to join the conversation.