SurgiScan

Logo
Dashboard
Pipeline
Aura

Inspiration

Every successful surgery starts with preparation, but even in top hospitals, 1 in 8 procedures are delayed or compromised due to missing or misidentified instruments. We spoke to scrub nurses, surgeons, and OR techs who described the stress of relying on memory and manual checklists in high-pressure environments. We realized this wasn’t a human failure, it was a systems failure. With AI, we saw an opportunity to build a second set of eyes in the OR: one that never forgets, never guesses, and never misses a critical tool.

What it does

SurgiScan is an AI-powered, head-mounted vision system that validates surgical tool layouts in real time. Before surgery, a nurse selects the procedure type. SurgiScan retrieves the standardized toolset from surgical databases. As instruments are placed on the back table, the wearable camera captures a live feed. Our CLIP-based model identifies each tool and semantically matches it to the procedure requirements, flagging missing, duplicate, or incorrect instruments immediately. It ensures the surgical team starts with confidence, not checklists.

How we built it

We developed a multi-stage pipeline starting with data collection: scraping and labeling thousands of surgical tool images, then augmenting and annotating them to improve recognition across angles and lighting. We fine-tuned a CLIP vision-language model to match tool images to natural-language descriptors from surgical databases. The system runs on lightweight edge devices with cloud fallback, enabling real-time inference. Our interface allows users to select surgeries, view validation results, and override or approve flagged tools, all with minimal training.

Challenges we ran into

Lookalike tools: Differentiating between similar instruments (e.g., Kelly vs. Crile forceps) required high-resolution data and bounding box tuning.
Lack of datasets: Medical tools are underrepresented in public datasets, so we built and labeled our own from scratch.
Real-time performance: We optimized our pipeline for latency, ensuring smooth operation on affordable hardware like GoPros or mobile devices.
UX in the OR: Designing a system that integrates into a fast-paced, high-stakes environment required deep research into surgical workflows and minimal touch interfaces.

Accomplishments that we're proud of

Built a working system that accurately detects and validates surgical tools from a live camera feed.
Surpassed 90% accuracy for core toolsets using only open-source models and no specialized hardware.
Developed a privacy-safe pipeline that captures no patient data, only tools, aligned with WHO surgical safety standards.
Created a scalable, cost-effective prototype with real potential to improve surgical outcomes globally, especially in resource-limited settings.

What we learned

Vision-language models like CLIP can be incredibly effective in medical tool recognition, especially when paired with semantic guidance.
Human-centered design is essential in healthcare, it's not just about what the model can do, but how teams interact with it under pressure.
Augmenting professionals, not replacing them, is the key to AI adoption in clinical settings.
Building trust in the OR starts with transparency, explainability, and seamless integration.

What's next for SurgiScan

Tool expansion: Broaden the model to recognize additional items like drapes, sutures, and devices that are often missed.
Inventory integration: Connect SurgiScan to hospital stock systems to track tool usage, location, and sterilization history in real time.
Smart hospital layer: Build toward a hospital-wide AI layer that manages and predicts instrument needs across departments.
Clinical testing: Launch pilot programs in Toronto hospitals to refine real-world performance and collect feedback from surgical teams.
Global impact: Adapt the system for low-resource settings by offering low-cost deployment kits and open-source models to hospitals worldwide.