-
-
2nd extended screen (for Surgeons)
-
Neuromonitorist screen
-
Inspiration
During complex brain and spine surgeries worldwide, surgeons rely on real-time neural signal feedback to prevent permanent nerve damage, yet traditional operating workflows face critical structural flaws. Interpreting raw signals introduces a dangerous 15–30 second communication delay between a nerve insult and surgeon awareness, while operating teams remain isolated from waveform data and unable to control software without breaking sterility. Compounded by severe shortages of specialized neuromonitoring staff, these delays and workflow barriers create unnecessary risks in high-stakes surgeries.
What it does
NeuroGuard is an autonomous, dual-display multi-agent assistant that transforms neuromonitoring into a surgeon-directed visual tool during brain and spine surgeries. Operating through a two-screen setup, it displays real-time, color-coded "traffic light" alerts and percentage signal drops on a second theater screen for surgeons, while allowing neurophysiologist to crop waveforms, log events, and run AI clinical troubleshooting on the main window. If neural signals drop below critical baseline thresholds, the system immediately speaks text-to-speech voice warnings over theater speakers, while surgeon voice or touch commands, drive hands-free, background cursor control on the primary IONM software without breaking sterility.
How we built it
Built using Python 3.11 and the Google Antigravity SDK, the platform orchestrates a multi-agent system (IONMSignalAgent, AmbientVoiceAgent, SurgeonAlertAgent, ClinicalTroubleshootingAgent) deployed on Google Cloud Run for sub-second, low-latency WebSocket synchronization between screens. Multimodal vision analysis is powered by Gemini 3.5 Flash via the Gemini API, which parses cropped waveform windows from native IONM software to calculate peak-to-peak amplitude drops against top-line baselines. The system features an interactive SVG vector muscle anatomy map (/surgeon) for real-time traffic-light alerts, integrates PyAutoGUI to execute surgeon voice/touch remote cursor automation on primary desktop software, and leverages ChromaDB with embedded Nihon Kohden user manuals for RAG-backed clinical troubleshooting.
Challenges we ran into
Building the system presented several complex engineering and clinical challenges. Engineering Gemini 3.5 Flash vision prompts to reliably isolate the top-line baseline waveform from active trace lines while filtering out background grid lines and 50/60 Hz electrical noise required precise bounding-box cropping and continuous prompt tuning. Additionally, achieving sub-second WebSocket state synchronization across two separate web views (/monitorist and /surgeon) was critical to eliminate browser rendering lag during rapid multi-agent executions. Finally, implementing OS-level background cursor automation with PyAutoGUI required exact coordinate mapping to ensure surgeon voice and touch triggers on Screen accurately manipulated native, legacy desktop intraoperative neuromonitoring (IONM) software without interfering with active user input.
Accomplishments that we're proud of
We are proud to have achieved a perfect pass rate across our automated agent verification pipelines, validating our multimodal vision parsing, audio event logging, and UI cursor automation. By converting complex, microvolt-level waveform data into an intuitive, color-coded traffic-light vector anatomy map, we made critical neural status readable at a glance for operating surgeons. Most importantly, we successfully bridged the clinical communication gap, reducing critical surgical alert latency from the traditional 15–30 second verbal delay down to sub-second visual updates and immediate spoken voice warnings.
What we learned
Building this project reinforced that AI agents in critical surgical environments must operate as active execution tools automating background OS controls and syncing live pipelines rather than passive chatbots. We also confirmed the necessity of a human-in-the-loop approach: AI must serve as an assistive co-pilot that enhances surgeon autonomy without removing clinician oversight. By combining real-time visual alerts with editable event logs, the platform drastically cuts operational friction while maintaining total safety and clinical authority.
What's next for NeuroGuard: Multi-Agent Neuromonitoring Workflow Automation
Next for NeuroGuard is finalizing HREC-approved multi-center observational study protocols across Australian hospitals under the TGA Clinical Trial Notification (CTN) scheme to validate system concordance against conventional IONM. We plan to package local Gemma models onto dedicated edge hardware consoles for completely offline, internet-free operation in shielded operating suites. Finally, we will develop predictive anesthetic triage algorithms that correlate depth-of-anesthesia (EEG burst suppression) and Train-of-Four (TOF) muscle relaxant fade with volatile gas levels, automatically suppressing false-positive alarms caused by anesthetic depth rather than true surgical injury.
