Surgent

AR Surgery Training with Biometrics & AI Surgeon Agent Consultation

Comment

Surgent

AR Surgeon Training with Biometrics & AI Surgery Agent Consultation

The Problem

In America, life is well with plenty of surgeons readily available but around the world

5 Billion people worldwide do not have access to surgical care (World Health Organization, 2019)

Resulting in 18 Million deaths every year because of a shortage of surgeons (World Health Organization, 2019)

18 million families, friends, and communities at the mournful mercy of something that can be prevented with more educated surgeons

So we followed this bloody trail asking why doesn’t the world have more surgeons?

And that’s because surgical training is limited.

We are talking about Expensive equipment, shortage of mentorship, and no one with a wound volunteering to be practiced on

This reflects a wide spread issue that while some nations are progressing fast in health,

There are countries in need being left in the dust,

So this is why we made Surgent,

Snap Chat’s first Augmented Reality Surgical Training system with Biometrics & AI Surgery Agent Consultation

Solution

1. Step-by-Step AR Surgery Guidance

The system overlays a step-by-step surgical workflow in AR, guiding the trainee through every phase of the procedure—from patient intake to incision, tissue handling, hemostasis, suturing, and post-operative care.

This reduces the ambiguity of “what to do next” and standardizes training across users.

2. Real-Time Patient Simulation

By integrating realistic patient roleplay and responses, the system allows trainees to interact with a simulated patient who can describe symptoms, express discomfort, or react to mistakes (e.g., incision errors, tool mishandling).

This adds clinical realism and builds communication skills alongside technical ability.

3. Proactive AI Mentor Coach

Unlike static tutorials, the AI Coach continuously observes trainee actions and intervenes with context-aware feedback:

Correcting poor incisions (“Your cut is too wide—try following the trace line”).

Preventing errors (“Don’t touch the organ edges, here’s why”).

Offering clarification on anatomy or next steps on demand.

This turns training into an interactive apprenticeship model rather than passive learning.

4. Performance-Based Metrics

Every session is scored with quantitative and qualitative feedback:

Time taken for each surgical stage.

Speed and precision of hemostasis.

Hand steadiness (tracked with motion/jitter data).

Correct vs. incorrect tool usage.

This creates a data-driven evaluation framework that helps both trainees and instructors measure progress over time.

5. Accessibility and Scalability

Because the simulation only requires AR glasses and lightweight AI software, trainees can practice anytime, anywhere—without needing a full operating room or expensive physical models.

This democratizes surgical training, especially for institutions with limited resources.

6. Agent Mail Integration

Delivers and tracks all of your results through email

Ask agent mail questions about the performance metrics

Recollect and recount the experience + learnings of the surgery procedure

7. Future-Ready Features

Gravity-based AR environments for realistic tool handling

Multi-user collaboration, where assistants and observers can join in real time

AI shadow explanations, where the mentor narrates the surgeon’s performance for students watching

How We Built It

We built Surgent as Snapchat’s first Augmented Reality (AR) surgical training platform, combining immersive interaction, AI mentorship, and biometric feedback.

Using Snap Studio, TypeScript, and custom 3D model generation, we created an experience that lets users practice surgeries with hand gestures and real-time feedback

Key components:

AR Surgical Toolset: Scissors, tweezers, gauze, anesthesia mask, needle & thread

AI Coach: A proactive advisor that gives real-time suggestions without needing prompts

Patient Roleplay: The simulated patient responds and interacts as part of the diagnosis and procedure

Performance Metrics: Tracks time per step, steadiness, incision quality, and more

Challenges We Ran Into

Achieving real-time, low-latency AI feedback inside AR environments

Simulating physical constraints like gravity and collision within Snap Studio

Balancing simplification with surgical realism

Building user trust in simulations as a valid form of training

Gen AI limitations: Our plan to use generative AI for dynamic responses didn’t work reliably, so we pivoted to writing custom logic and branching narratives

Lighting issues: Showing 3D models clearly in AR required troubleshooting lighting schemes in the environment

Gemini Live view limitations: We tried to use Gemini Live to show only the augmented world for demos, but were blocked by camera blending restrictions

Accomplishments That We're Proud Of

Built a full AR surgery sim using only a phone—no extra hardware

Created a smart AI Coach that provides real-time feedback as users work

Simulated real-time patient dialogue and interaction

Laid the groundwork for biometric performance evaluation and potential certification

What We Learned

AR can powerfully simulate and teach hands-on skills like surgery

Users love real-time, unprompted feedback from AI—it’s like having a coach in your pocket

Immersive storytelling increases engagement and retention

Blending software, education, and design requires fast, flexible iteration

What's Next for Surgent

Physics-Based Interactions: Add gravity and object placement requirements

Multiplayer Mode: Let surgeon + assistant train together or shadow each other

Mentor Mode: Allow mentors to watch and receive AI-generated performance notes

Snap Spectacles Integration: Let patients see the surgery on themselves through AR glasses

Expanded Metrics: More detailed scoring, error detection, and biometric tracking

Built With

+ 18 more
Share this project:

Updates