AI Decision Governance System

Inspiration

AI in healthcare has made huge strides when it comes to predicting patient outcomes and recommending treatments. Still, there's one big issue: AI can't act on its own without someone keeping an eye on it. Right now, you either have a doctor double-checking every AI output, or you let the system run wild—which is risky if the training wasn’t perfect or something weird pops up in the data. We wanted to fix that. Our goal was to create a system that checks both AI predictions and the underlying sensor data before anything happens, making sure it's safe to let AI act without a human hovering over it.

What it does

Our project is basically the gatekeeper between AI recommendation engines and actual medical devices. Think of it as an "Autonomy Permission Engine." It doesn’t just take AI predictions at face value. Instead, it looks at several things: how confident the AI is in its prediction, whether the sensors are working reliably, how stable the underlying causes seem to be, and how much the current patient looks like the ones the AI actually studied.

From all that, it calculates a permission score—a kind of eligibility rating for each intervention. If the score’s high enough, it lets the AI take over and execute the clinical action without human input. If it’s not, it hits pause, alerts the medical staff, and waits for someone to approve the intervention.

How we built it

The prototype is made up of interconnected software modules.

The Data Gathering Module grabs real-time readings like heart rate, blood pressure, oxygen saturation, and respiration rate from patient monitors. The AI Prediction Module runs machine learning models (built in Python) that analyze this data, flag possible health risks, and make recommendations. For the Sensor Reliability & Uncertainty Estimation part, we built tools that inspect the sensor data’s stability and work out how trustworthy the AI’s predictions are.

All these pieces come together in the Permission Engine. It crunches the numbers from every module, constantly updating the safety score. We also built a web interface so doctors can see scores and sign off on anything that needs an extra check.

Challenges we ran into

Trying to measure something as fuzzy as “trust” isn’t simple. Sorting through sensor data to spot noise or faulty readings was a real challenge. Plus, getting the AI’s confidence level right so it actually reflects real-world safety took some fine-tuning. Making sure our permission score meant what we wanted was probably the hardest part.

Accomplishments that we're proud of

We managed to build a governance layer that actually works as a bridge between AI and real clinical practice. It filters out unreliable decisions, saves doctors from pointless manual checks, and blocks unsafe automation. Seeing the system do exactly what it’s supposed to is pretty gratifying.

What we learned

Accuracy isn’t everything. You can have the best predictions in the world, but if you can’t guarantee safe execution, it’s not enough. Putting this system together taught us how to weave uncertainty-aware AI and sensor checks into one smart controller.

What's next for the project

The core architecture is done, so we're moving to file a provisional patent to lock in our priority date. After that, we'll refine the prototype and start thinking about how we can get it into hospitals, telehealth platforms, and wearable devices. Commercialization is definitely on our minds.