UMDE: AI-Powered CMS Rule Extraction and Validation

1. Inspiration

While designing a Risk Adjustment submission tool, I faced a challenge that had little to do with engineering tools — and everything to do with interpreting CMS.

The Centers for Medicare & Medicaid Services (CMS) do provide documentation — including the EDPS Submission Guide, Edit Rules Guide, and various file layout specifications — but these resources are spread across multiple formats like PDFs, Excel spreadsheets, and supplemental guides. There’s no central API, no machine-readable schema, and no canonical rule set to directly plug into a validation engine.

To understand how CMS validates a member record, you have to read through these documents line by line, interpret what the edits imply, and manually translate that into software logic.

That process takes time. It’s fragile. And every health plan has to repeat it — often reinventing the same validation logic in slightly different ways, just to stay compliant.

This led me to ask: "What if AI could take on the heavy lifting — reading CMS documents, identifying member-level validation rules, and structuring them into usable code?"

That was the seed idea behind UMDE: to reduce the manual burden of CMS compliance by applying deep learning and transformer models to rule extraction.

But there was another factor: cost.

Many existing tools come with licensing or integration costs that may not be easy to justify — especially for teams who just want to check their data before submission. My goal was to build something more approachable — something health plans could start using without a sales process, a contract, or a long wait.

UMDE is being built with that principle in mind: open by default, and accessible from day one. If someone finds value and wants to scale, there can be a sustainable subscription model — but the threshold to participate should be low.

Finally, I considered what happens when CMS updates its documentation — which they do regularly. Rules change. Field formats evolve. And health plans are expected to notice, reimplement, and validate all over again.

Instead of repeating that burden, I envisioned something better: Let the health plan admin upload the new CMS document, trigger the extraction engine, and review any newly detected or modified rules.

And here’s where something important comes in: AI doesn’t replace the human — it assists them.

Even with a multi-model AI pipeline running extraction and comparison logic, UMDE is designed to keep a human in the loop. The health plan analyst or admin will always have the final say — to review, approve, or reject each proposed rule.

The goal isn’t full automation. It’s about reducing friction, increasing visibility, and helping every health plan stay compliant without burning cycles on document archaeology.

That’s what inspired UMDE: A system that empowers health plans to understand, test, and evolve with CMS — not chase it.

2. What It Does

While the broader vision for UMDE is still in progress, the backend already delivers a working core: a system that reads CMS documents, extracts relevant rules using AI, and generates real validation logic — all with a human in the loop.

Here’s what’s already built:

1. Reads CMS Documents in Multiple Formats

   • Supports both PDF and Excel input, reflecting the way CMS actually publishes its documentation.

2. Extracts Rules Using a Multi-Model AI Engine

   • Uses DeepSeek, Zephyr, FLAN-T5 (future fine-tune target), and a locally hosted Gemma model — with a final fallback to a non-AI keyword-based extractor.

3. Selects the Best Candidate Based on Confidence

   • Automatically compares outputs and surfaces the highest-confidence rule for human review.

4. Approved Rules Are Structured for Reuse

   • Approved rules are stored in a consistent structure for downstream use.

5. Each Rule Is Classified by Complexity

   • Auto-tagged as Level 1–4 (from simple format checks to complex conditional logic).

6. Simple Rules Can Auto-Generate Validation Logic

   • L1–L2 rules can trigger auto-generation of Python-based validation logic for integration into the engine.

3. How We Built It

• Modular fallback engine for AI model orchestration
• Parsers for both PDF and Excel CMS files
• Confidence scoring logic to choose the most trustworthy model output
• Human-in-the-loop flow with approval gating
• Structured rule storage format with field mapping and metadata
• Automatic rule complexity classification
• Validation logic generator for simple rules
• Angular frontend partially scaffolded (static page for now)

4. Challenges We Ran Into

• Avoiding infinite fallback loops and retry traps
• Finding a free model that works well for CMS language (Zephyr = usable, Gemma = still untested)
• Couldn’t yet fine-tune FLAN-T5 due to time limits
• Parsing inconsistent CMS formats (PDFs, Excel)
• Confidence scores vary wildly across models
• Early bugs in non-AI fallback registration
• Limited test coverage on large-scale validation
• Frontend not yet integrated

5. Accomplishments We’re Proud Of

• Full backend rule extraction pipeline is working
• Modular fallback system with confidence-based selection
• Structured traceability for each rule back to its CMS source
• Logging across model attempts and fallback transitions
• Human-in-the-loop approval pipeline
• Automatic complexity classification (L1–L4)
• Once approved, simple rules (L1–L2) can be turned into working validation logic automatically — with no manual coding required. This allows health plans to incorporate updated CMS rules with zero development effort.
• Test-driven development foundation in place
• Test-driven development foundation in place

6. What We Learned

• AI requires normalization, not just model chaining
• Extraction needs tight guardrails and confidence logic
• Structured rule design early on pays off later
• Human reviewers are critical for trust in compliance logic
• “Error-free build” isn’t enough — real-world testing is everything

7. What’s Next for UMDE

While the backend foundation is in place, UMDE is still early in its journey. Several important capabilities are already designed — but not yet implemented. These represent the logical next steps for turning the engine into a production-ready, end-to-end platform.

1. Fine-tune FLAN-T5 into a CMS-aware model
   The infrastructure is in place — the next step is to fine-tune the FLAN-T5 model using CMS documents to improve precision and reduce hallucinations.

2. Introduce DAG-based rule execution
   Shift from sequential rule evaluation to dependency-aware DAG execution for more efficient and accurate rule processing.

3. Support CSV and FHIR data ingestion
   Extend file upload support to handle FHIR JSON bundles alongside CSV files — normalizing both formats into a unified record model.

4. Complete the upload flow and field mapping logic
   Finalize backend integration with file upload, field detection, and alias mapping based on configuration files.

5. Add database, Redis, and AWS integration
   Persist uploaded data, extracted rules, and results using relational storage; use Redis for rule caching; integrate with AWS for deployment, logging, and S3-based document retention.

6. Build frontend dashboard with rule viewer and approval flow
   Complete UI for uploading files, reviewing extracted rules, approving them, and viewing validation outcomes.

7. Show users which rows failed validation and why
   Include a UI component to display failed records, highlighting the specific rule violated, the incorrect value, and the expected result.

8. Provide an interface to correct data and re-run validation
   After viewing errors, users should be able to edit records directly in the interface and re-submit them for validation without starting over.

9. Add diff-aware CMS document comparison
   Allow users to upload a newer version of a CMS document and automatically detect which rules are new, modified, or removed.

10. Build audit dashboards and rule-level analytics
    Visualize metrics like rule confidence, pass/fail trends, fallback usage, and historical rule acceptance.

11. Add longitudinal member view for high-risk EDPS insight
    Create a longitudinal view that stitches together member data across multiple submissions, focusing on the top 10 EDPS rule fields that are most commonly associated with encounter rejections. This will help analysts detect patterns, trends, and chronic data gaps that may otherwise go unnoticed.

12. Rope in health plans to use UMDE and save millions of dollars.

Privacy and Production Hardening

The eventual goal is to make UMDE compliant with:

• HIPAA
  
• HITRUST
  
• ISO 27001
  
• ISO 27701
  

This will guide future work on security, auditability, and deployment standards.

Built With

• angular.js
• python