Inspiration

it was 4 AM when the nurse rushed into the residents' room "Doctor, a 59-year-old COPD patient, just came in. This isn't his first exacerbation. He has an arrhythmia, pH below 7.25, and he is unconscious" the junior resident admitted the patient, intubated him, and stood in front of the ventilator. He had never managed one alone before, he tried calling a senior resident. Then a consultant. Nobody answered. It was 4 AM. The patient's family was outside, his wife and three daughters, visibly from a low-income household, their father the sole breadwinner, pressing against the ICU doors wanting to know if he would survive.

with no backup, no mentor, and a ventilator waiting for values, the resident did what any of us would do under that kind of pressure he opened a textbook on his phone and spent 3 minutes searching for the right settings. twelve minutes had passed since intubation. He found the settings, and entered them. He thought he had done his job.

what he forgot, what nobody had ever properly taught him, was to monitor tidal volume, plateau pressure, and driving pressure in real time. The pressure in that patient's lung exceeded what that tissue could bear. A barotrauma developed: the lung ruptured, air accumulated in the chest cavity, and a cascade of complications followed that no amount of intervention could reverse.

The patient died the next morning.

I was an intern that night. On my way out, his wife stopped me at the door. With exhausted eyes she asked:" will he survive ?" I told her I was just an intern and didn't know. I walked home unable to shake the feeling that something had gone catastrophically, preventably wrong.

two weeks later, I was rounding with an ICU consultant when we approached a mechanically ventilated patient whose settings were clearly off. The consultant read the waveform in seconds, identified the issue, explained it to us, corrected it. Same machine, same parameters. completely different outcome, because one person in that room knew exactly what to look for.

the contrast between those two moments never left me. I decided to fill my knowledge gap. I searched YouTube, and found only fragmented, low-quality videos that barely scratched the surface. I eventually enrolled in a paid Mechanical Ventilation fundamentals course. I was not alone: 68 senior physicians registered alongside me. by the end , only 2 were actively participating.

In a single course, the knowledge gap I had witnessed at 4 AM reproduced itself at scale, not among interns, but among experienced clinicians who had presumably been managing ventilators for years.

I turned to Notebook LM's deep search, which gave me the most comprehensive, structured overview of MV I had encountered, and for the first time I began to truly understand ARDS, COPD exacerbations, and ventilator weaning at a mechanistic level. The knowledge was out there, the problem was never access it was delivery, timing, and format.

That resident's experience is not an exception, it is the norm and the literature proves it. Mechanical ventilation is among the most important skills in critical care, and among the most poorly taught. Nearly 50% of trainees report dissatisfaction with their MV education, and of 15 studies examining MV training, only 2 ever measured its impact on actual patient outcome [1]. The failure runs deep as a 2024 mastery assessment found that 20% of attending intensivists failed, only 12% of residents correctly identified life-threatening asynchronies, and mean proficiency scores were just 6.2/10 across all levels, even as a global intensivist shortage forces non-specialists into ventilator roles they were never trained for [2,3]

The death toll this gap produces is precisely quantified. I n a cohort study of 1905 ICU patients, 40% received tidal volume above 8 ml/kg PBW in direct violation of Grade 1A evidence, and those exposed for over 24 hours faced 82% higher odds of in-hospital death [4]. Every 1ml/kg increase in tidal volume at ARDS onset carries a 23% increase in ICU mortality risk [5], meaning the most dangerous decision is often the very first one. A 2025 review further confirmed that failing to minimize driving pressure below 15 cmH2O from the onset of ventilation independently increases 30-day mortality [6], yet this single calculation is routinely skipped at the bedside.

these are not academic statistics; they are the story of that patient's family standing at the ICU door at4 AM, they are preventable deaths. VentLive AI was built to fill this gap and calculate all the required values to improve patient's quality of care.

On February 22nd, my professor mentioned the Gemini Live Agent Challenge. For the first two days, I ignored it, I had no original idea, no coding background, and no clear problem to solve. Then I thought about the ventilator. The problem had been in front of me the whole time: an exhausted junior physician, alone, at 4 AM, holding a phone-sized textbook open with one hand while managing a critically ill patient with the other. No real-time decision support, no structured memory of what had been tried, no voice to speak to. just silence, pressure, and a machine waiting for numbers.

That was the idea that was the app.

What it does

VentLive AI is a real-time, voice-interactive clinical decision support agent for ICU physicians managing mechanically ventilated patients.

A physician speaks naturally, or type in noisy environments and VentLive AI responds immediately with a fully structured SCCM-guideline-grounded assessment covering five clinical scenarios: ARDS, COPD, ARDS/COPD overlap, Weaning, and Post-Operative SBT.

under the hood, two Gemini models work in tandem:

  • Gemini Live 2.5 Flash Native Audio handles real-time bidirectional voice: receiving the physician's speech as a continuous audio stream, managing natural turn-taking, and delivering spoken clinical assessments through a natural voice. Critically, the Live model operates under a hard constraint, when the physician speaks, it says only "Analyzing." and waits. It never generates clinical content from its own knowledge. Every word it speaks is verified text injected from the reasoning engine after the full assessment is complete. When a physician asks a follow-up question during a voice conversation, the answer is constructed exclusively from the fields of the most recently verified SCCM assessment card, driving pressure value, P/F ratio, next steps, escalation criteria, not from the model's parametric knowledge. This eliminates the hallucination risk at the point where it is most dangerous: the spoken clinical recommendation a physician acts on immediately.

  • Gemini 2.5 Flash handles clinical extraction: receiving the physician's transcript alongside the patient's full clinical history, extracting structured ventilator parameters, ABG values, hemodynamics, and weaning indicators from natural speech, including voice recognition artifacts, mixed Arabic and English ICU terminology, and noise tags. When the Gemini extraction API is unavailable, 22 purpose-built regex extractors covering every clinical field activate automatically, and the full reasoning engine continues operating on the extracted values. The system never goes silent.

  • This architecture enforces a deliberate sequence: the complete evidence-based clinical assessment finishes first, then the result is injected as spoken text to the Live model. The physician hears a complete, structured answer every time, never a half-formed one generated mid-reasoning.

  • VentLive AI automatically calculates all critical bedside metrics in real time, the values most commonly skipped under pressure: driving pressure, P/F ratio, S/F ratio, TV/IBW ratio, RSBI, and MAP. S/F ratio is used as a surrogate when ABG is unavailable and SpO2 less than 97%

  • Based on these values and the full clinical picture, VentLive AI also displays a real-time SBT Readiness Assessment, synthesizing oxygenation, hemodynamics, RSBI, and clinical criteria to indicate whether the patient is a candidate for a spontaneous breathing trial.

  • The SCCM Assessment Card- The SCCM card is the visual centerpiece of VentLive AI, and every designs decision was made with one goal: allowing a physician under pressure to understand the patient's status in under three seconds.

    • color-coded status system: the entire card is color-coded by clinical severity: green for stable, orange for worsening, and red for critical case.
    • Trend badges: from the second assessment onward, trend badges appear automatically for Pao2, Ph, PEEP, and MAP. showing at a glance whether each value is improving, stable, or deteriorating relative to the previous entry. No scrolling through history required.
    • Numbered, color-coded immediate next steps: each action is numbered and color-coded by urgency, so the physician knows not just what to do, but in what order and how urgently.
    • Safety flags: A dedicated red alert box renders at the top of the card, but only when clinically meaningful flags are present. If there are no active safety concerns, the box does not appear. No alarming fatigue from empty warnings.
    • bulleted, color-coded escalation & monitoring sections: each criterion is individually color-coded, making it immediately clear which monitoring parameters are most time-sensitive and which escalation triggers are closest to being met.
    • smart card rendering: the SCCM card does not appear on launch. It renders only after the first clinical assessment is complete, and updates only when new clinical data is provided, preventing stale, and overcrowded cards from misleading decision.
    • Context-aware staleness detection. If new ventilator settings are reported without accompanying ABG results, VentLive AI detects the mismatch and alerts the physician with a yellow warning banner. Staleness thresholds are context-dependent: 30 minutes after any ventilator change, 2 hours for Critical or Worsening patients, and 8 hours for Stable patients, reflecting the clinical reality that a stable patient's 6-hour-old ABG is informative, while a deteriorating patient's 2-hour-old gas is already stale. Acting on outdated ABG values after mechanics have changed is a common and dangerous ICU error. VentLive AI catches it before the recommendation is generated.
    • ARDSNet PEEP/FiO2 Table Integration: In a real ICU at 4 AM, no one has the ARDSNet table memorized. The moment a clinician reports Fio2 and ARDS severity, VentLive AI instantly cross-references the full ARDSNet protocol table, identifies the gap between the current and recommended PEEP, and flags which table applies based on severity level, preventing the common error of applying the wrong table to the wrong patient. Critically, it also attaches a guideline uncertainty note where ATS and ESICM recommendations diverge, so the clinician can individualize the decision with full awareness of the evidence landscape. every PEEP adjustment recommendation is accompanied by an automatic reminder to recheck driving pressure and Pplat immediately after, closing the loop that is most commonly left open under pressure.
    • Guideline citation transparency: Every clinical action in the SCCM card carries its source guideline inline, ATS 2024, ESICM 2023, GOLD 2024, ATS/ACCP 2017, AARC 2024, PROSEVA, EOLIA, so the physician knows not just what the system recommends, but on what evidence and from which authority. The card also shows whether the assessment reflects the most current data or is based on a previous update, making the currency and basis of every clinical decision fully traceable at a glance.

Every response is structured as:

  • ventilation status: stable/worsening/critical
  • physiological interpretation: what the numbers mean clinically
  • immediate next step: numbered and color-coded
  • monitoring & safety checks: bulleted and color-coded
  • escalation criteria: bulleted and color-coded
  • critical metrics: driving pressure, P/F ratio, S/F ratio, TV/IBW ratio, RSBI (weaning only), MAP. each with severity color-coding and target reference.

Every line of clinical logic was validated against ATS 2024, ESICM 2023, GOLD 2024, and AARC guidelines before deployment.

How we built it

I began planning the architecture: which diseases to cover, what the AI output should look like, how the voice interaction should feel, how case memory should persist. I initially intended to use Claude Code inside Google Antigravity, but due to the limitation in my country I couldn't purchase the subscription for Claude Code, and my laptop would freeze when I run Google Antigravity. the financial and hardware constrains made it impossible, so I settled on the free version of Claude paired with Google Colab, and made it work.

the first version was built entirely through prompting. I wrote zero lines of code myself, copying and pasting Claude's outputs into Colab cells, watching the app comes to life one block at a time. For the first time , I witnessed what AI-assisted development actually feels like, and it was extraordinary.

But the first two versions were messy. The code was tangled, hitting Claude's limits multiple times daily, and I was terrified of accidentally deleting a cell that would unravel everything. I was overwhelmed, and I knew it. So I stopped and rethought the approach entirely. In the third version, I restructured everything: Claude projects for organized context, dedicated single chats for isolated problems, 'sed' and 'grep' for targeted edits instead of full rewrites, and a handoff protocol when approaching Claud's limits, asking Claude to summarize everything needed to continue in a fresh session, then continuing from there. The code became clean, modular, and maintainable.

Tech Stack: Backend Language: Python 3 Frontend Languages: JavaScript (ES6+), HTML5, CSS3 Web Framework: FastAPI + Uvicorn Cloud Platform: Google Cloud Platform AI Models: Gemini 2.5 Flash, Gemini Live 2.5 Flash Native Audio AI Platform: Vertex AI Database: Google Cloud Firestore (+ in-memory fallback) Real-Time Comms: WebSocket, Web Audio API, AudioWorklet Auth & Security: API Key, CSP, security headers, HTML sanitization Dev Infrastructure : ngrok (HTTPS tunnel) AI SDK: Google GenAI Python SDK

Challenges we ran into

When I started reading the competition rules and quickly realized I was facing a steep learning curve, I had never heard of live agent, had no idea what Gemini live API, and ADK was completely foreign concept. I began searching, reading, watching, and absorbing as much as I could in a compressed window of time. I have always been a technology enthusiast, someone who followed every major AI development and always asked myself "How can this help me as a physician ?" But I had never built an application from scratch. This was new territory.

Building a real-time clinical AI agent on a tight deadline. with no formal software engineering background, produced challenges that were both technical and clinical.

  • Negation blindness in clinical Natural Language Processing (NLP) was the first major obstacle: early versions parsed "the Spontaneous Breathing Test (SBT) was not tolerated" as a successful spontaneous breathing trial. I solved this with three layered defenses in the weaning branch of the reasoning engine. First, a dedicated _is_negated() function checks a 50-character window before each matched keyword for 20+ negation patterns , evaluating every keyword in its own negation context, so "patient has NOT failed the trial" never triggers the failure path while "patient failed the trial" correctly does. Second, a structured sbt_status field extracted by the AI and regex pipeline provides categorical values , "completed", "passing", or "failing" , that supplement keyword matching: when keyword detection is inconclusive, the structured field resolves the ambiguity; when both agree, they reinforce each other. Third, a full-support guard prevents numeric signals like high respiratory rate or low SpO₂ from being misread as SBT failure when the patient is still on full ventilatory support and no trial has started.

  • Turn state corruption in the Gemini Live API emerged when speech injection caused the session to stop responding mid-conversation. The fix required careful management of injection timing and method selection: all clinical assessment text is now injected via a single session.send(input=text, end_of_turn=True) call with a settling delay before transmission, with automatic fallback to send_realtime_input() if the primary method fails. Failed injections are preserved in a pending_speech_inject buffer and re-delivered automatically on the next session reconnect, ensuring no clinical assessment is lost across the Gemini Live session lifecycle.

  • API timeouts under load required implementing a retry wrapper with three attempts, a 20-second per-call timeout, and exponential backoff at 1, 2, and 4 seconds between attempts. When all three attempts fail, the system activates graceful degradation: all 22 regex-based clinical field extractors run directly on the raw doctor input, and the extracted values feed the full SCCM reasoning engine, producing a real clinical recommendation with a degradation warning rather than silence. A clinical tool that goes silent during an ICU emergency is worse than no tool at all.

  • Model availability misinformation cost significant development time early in the project. AI-assisted research incorrectly indicated that Gemini 2.5 flash live was unavailable in Egypt, and at that stage of my technological knowledge, I had no way to verify it independently. I did not yet know how to check regional availability or test model endpoints directly. I simply trusted the answer, the invisible tax non-developers pay. This forced a fallback to Gemini 2.0 flash live, and to compensate for its weaker speech recognition I built an extensive map parried with Latin-script transliteration map, teaching the system how ICU physicians actually pronounce common ventilator related terms in both Arabic and accented English. By the time I discovered 2.5 was available and upgraded, the layer was too deeply integrated to remove, and too useful to want to. The hours spent on workaround produced something the original design never planned for: a bilingual ICU voice agent that works in Arabic and English simultaneously.

  • Tablet portrait mode, the most common bedside device orientation, caused the chat input to disappear off-screen due to two compounding issues. The first was Android Chrome's dynamic toolbar: 100vh in CSS includes the browser toolbar height, so when the toolbar auto-hides on scroll the input area is pushed below the visible viewport. This was fixed with a JavaScript --vh custom property that recalculates on every resize and orientation change using window.innerHeight, which always reflects the true visible area. The second was a flexbox minimum-height collapse: the chat scroll container refused to shrink below its content height, displacing the input area downward. This was resolved by structuring #main as a fixed-height column flex container with overflow: hidden, giving #chat flex: 1 with overflow-y: auto to scroll internally, and marking #status-bar with flex-shrink: 0 so the input area is always anchored at the bottom of the visible screen..

  • The ARDS/COPD overlap required the most careful clinical thinking: when a patient has both conditions simultaneously, PEEP optimization goals directly conflict, ARDS demands higher PEEP for alveolar recruitment while COPD demands lower PEEP to prevent dynamic hyperinflation. I implemented a dedicated overlap branch that runs both ARDS and COPD constraint sets simultaneously rather than choosing one over the other. When auto-PEEP is detected, the ARDSNet PEEP table recommendation is explicitly overridden and COPD hyperinflation controls, RR ceiling of 12 per minute, external PEEP capped at 75% of measured auto-PEEP, are inserted at the top of the action list before any ARDS lung-protective ventilation steps. When P/F falls below 150, ARDS rescue interventions, prone positioning and neuromuscular blockade, are added on top of the active COPD constraints rather than replacing them. A single escalation output carries mandatory criteria for both conditions simultaneously: ECMO referral thresholds from the EOLIA trial for the ARDS component and the PaCO₂ rapid-correction safety limit from GOLD 2024 for the COPD component, ensuring neither condition's danger signals are silenced by the other.

  • VentLive AI stores each COPD patient's chronic baseline PaCO₂ at admission, monitors every subsequent ABG against that individual floor rather than the textbook normal, and alerts the physician in real time if mechanical ventilation is correcting CO₂ faster or further than the patient's physiology can safely tolerate — preventing the post-hypercapnic metabolic alkalosis, cerebral vasoconstriction, and ventilator dependency that over-correction reliably produces.

  • Building under impossible constraints was perhaps the most honest challenge of all. I am a practicing physician. This project was built entirely in the margins, nights that stretched to 13 hours straight, sessions that started after a full clinical shift and ended at dawn, knowing the next day belonged entirely to patients and not to code. There was no luxury of iteration time. Every session had to count, every problem had to be solved before the alarm went off. That pressure, ironically, is exactly what the tool is designed to eliminate for the next junior resident standing alone in front of a ventilator at 4 AM.

Accomplishments that we're proud of

Shipping a functioning, clinically validated real-time AI agent with zero software engineering background. built entirely through prompting, persistence, and post-shift nights. is the accomplishment this project represents at its core.

But beyond the personal: VentLive AI correctly handles five clinical scenarios ( ARDS, COPD, ARDS/COPD overlap, Weaning failure, and Post-Operative SBT), calculates driving pressure in real time, maintains persistent structured case memory across patients, and delivers voice-interactive SCCM-guidelines-grounded recommendations, all validated against current ATS, ESICM, GOLD, and AARC guidelines before a single line reached production.

A physician who had never written a function built the tool that should have existed the night that patient died.

That is what I am proud of.

What we learned

This project changed how I think about three things simultaneously: AI, medicine, and myself.

Clinically, I learned that ARDS, COPD, and Weaning failure are not just different diseases, they represent fundamentally different physiological states with competing ventilation goals that cannot be managed by a single algorithm. The nuance lives in the overlap, the timing, and the trend. exactly what a persistent structured case memory is designed to capture.

technically, I learned that building a real-time AI agent is less about writing code and more about designing the right architecture before writing a single line. The difference between version one and version three was not more code, it was better thinking before the code. Constraints, it turns out, are not obstacles to building well. They are the reason you are forced to.

Personally, I learned that the gap between "I have an idea" and "I have a working product" is bridgeable, but only if you are willing to be uncomfortable, ask for help, fail in small controlled ways, and keep going.

This process taught me something I had not expected: AI does not eliminate the need to understand what you are building, it accelerates the learning curve for those willing to engage with it seriously. Every bug that appeared after a fix forced me to understand the system more deeply. I took two Python courses and an HTML course back in 2020, remembered almost none of it. but working through real problems with AI guidance rebuilt that knowledge in context, which is the only way it actually sticks.

That patient's family at the ICU door at 4AM deserved better than a resident searching a phone-sized textbook alone. VentLive AI is the tool that should have been there.

What's next for VentLive AI

VentLive AI was built as an MVP under severe time constraints, but the clinical vision extends well beyond this version.

It is worth being transparent: two features that were part of the original design (ventilator screen image analysis and the undifferentiated respiratory failure module) were not implemented in this submission. Not because the ideas were abandoned, but because the reality of building this project as a practicing physician, across stolen night hours and post-shift session, made it impossible to execute them to the clinical standard they deserve within the submission window. They remain the two feature I am most committed to building next.

  1. Ventilator Screen Image Analysis A physician photographs the ventilator display, and VentLive AI reads the values directly, no manual entry required. This makes the tool usable in the most chaotic moments of an ICU shift. This is a natural extension of the Gemini multimodal capability already powering the platform.

  2. Undifferentiated Respiratory Failure Module Not every intubated patient arrives with a clean diagnosis. This branch will handle the most dangerous scenario in the ICU: a ventilated patient with no established diagnosis. VentLive AI will guide the physician through an evidence-based diagnostic pathway, recommending initial safe ventilator settings based on available data, suggesting the most likely differential , and walking through advanced life support protocol in parallel. This transforms VentLive AI from a condition-specific tool into a true first-responder clinical agent.

  3. Trend Intelligence and predictive alerts with persistent case memory already in place, the next logical layer is pattern recognition across time, detecting deterioration trajectories before they become crises, flagging when a patient's driving pressure trend predicts barotrauma risk, or when ABG trends signal impending weaning failure.

  4. Institutional partnerships and development the long-term goal is integration into ICU workflows at teaching hospitals, particularly in settings where intensivist coverage is limited and junior physicians are routinely making high-stakes ventilation decisions alone and without support.

VentLive AI was designed by a physician, for physicians. Built from the memory of a patient who deserved better, and every future patient who still does.

References:

[1] Keller JM, Claar D, Ferreira JC, Chu DC, Hossain T, Carlos WG, Gold JA, Nonas SA, Seam N. Mechanical Ventilation Training During Graduate Medical Education: Perspectives and Review of the Literature. ATS Scholar. 2020.

[2] Hayashi FK, Sousa MLA, Garcia MVF, Macedo BR, Ferreira JC. Simulation-based Assessment to Measure Proficiency in Mechanical Ventilation among Residents. ATS Scholar. 2022.

[3] Amirahmadi R, Seam N, Lee BW. Persistence and Completion Rate of a Year-long Mechanical Ventilation Preceptorial Education Course Among Critical Care Fellows and Attending Intensivists (PIPE Study). American Journal of Respiratory and Critical Care Medicine. 2025.

[4] Sjoding MW, Gong MN, Haas CF, Iwashyna TJ. Evaluating Delivery of Low Tidal Volume Ventilation in Six ICUs Using Electronic Health Record Data. Annals of the American Thoracic Society. 2019.

[5] Needham DM, Yang T, Dinglas VD, Mendez-Tellez PA, Shanholtz C, Sevransky JE, Brower RG, Pronovost PJ, Colantuoni E. Timing of Low Tidal Volume Ventilation and Intensive Care Unit Mortality in Acute Respiratory Distress Syndrome: A Prospective Cohort Study. American Journal of Respiratory and Critical Care Medicine. 2015.

[6] Merola R, Vargas M, Battaglini D. Revisiting Acute Respiratory Distress Syndrome Ventilation Management: Time for a Paradigm Shift Focusing on Tidal Volume. 2025.

  • ATS 2024 Clinical Practice Guideline — Adult ARDS Management
  • ESICM 2023 ARDS Guidelines Summary
  • GOLD 2024 — Global Initiative for Chronic Obstructive Lung Disease
  • ATS/ACCP 2017
  • AARC Clinical Practice Guideline — Spontaneous Breathing Trials
  • ARDSNet PEEP/FiO₂ Tables
  • EOLIA Trial Criteria (ECMO in severe ARDS)
  • VUMC Multidisciplinary Surgical Critical Care Service Protocols
  • Evidence-Based Mechanical Ventilation Management — Comprehensive Clinical Report (ARDS, COPD, Liberation)
  • Deranged Physiology — Optimal PEEP & Extubation Readiness
  • PMC: PEEP Management in Prone Positioning & ECMO
  • PMC: Respiratory Care Management of COPD Exacerbations
  • PMC: NIV in COPD — When, Where, How to Start and Stop
  • PMC: NIF Thresholds for COPD Extubation Readiness
  • PMC: ScvO₂ During SBT as Extubation Predictor
  • Dr. Oracle — Initial MV Settings for COPD/Pneumonia & ARDS
  • IEMP 2024 — Mechanical Ventilation (Emergency Medicine)
  • PMC: Management of Asthma and COPD Exacerbations in the ICU

Built With

  • aarc-2024
  • api-key-authentication
  • ardsnet-protocol
  • ats-2024
  • ats/accp-2017
  • audiocontext-16khz
  • audiocontext-24khz
  • audioworklet
  • berlin-definition
  • content-security-policy
  • cors-middleware
  • css3
  • eolia-trial
  • esicm-2023
  • fastapi
  • gemini-2.5-flash
  • gemini-live-2.5-flash-native-audio
  • gold-2024
  • google-cloud
  • google-cloud-firestore
  • google-cloud-project
  • google-colab
  • google-genai-python-sdk
  • html-sanitization
  • html5
  • javascript-es6+
  • ngrok
  • proseva-trial
  • pydantic
  • python
  • rest-api
  • rice-2007
  • security-headers
  • unicorn
  • vertex-ai
  • web-audio-api
  • websocket
  • websocket-api
Share this project:

Updates

Submission history