Helpit

A memory-grounded co-pilot for robotic dental surgery. Dentists get full patient history, cross-patient case recall, and live deviation alerts — all before the robot ever touches a tooth.

Comment

HELPIT — Memory Agent for Robotic Dental Surgery

Inspiration

Picture the second right before a robotic arm lowers a drill bit toward a patient's jaw. It has sub-millimeter precision. It has a plan, loaded and ready. What it doesn't have is any idea that this same patient's last implant failed on the other side of their mouth three years ago.

That's not a hypothetical. Robotic dental implant surgery is no longer science fiction — implant-placement robots are already drilling and seating implants in real clinics, right now. But precision was never the missing piece. Memory was.

Every one of these systems starts every single procedure the exact same way: cold. The robot has no idea this patient's bone density scan looked different six months ago than it does today. It has no idea the last patient with this exact bone-density class and this exact implant brand had a 30% higher rejection rate the moment torque crossed 45 Ncm. That knowledge lives in one dentist's memory, in a paper chart nobody cross-references mid-procedure, or nowhere at all — a robot with world-class precision and the situational awareness of a stranger.

That gap felt uncomfortably familiar. It's the same gap the WHO Surgical Checklist was invented to close in human-led surgery — and even there, under real time pressure, checklists get skipped and critical steps go undocumented. Only 23.1% of laparoscopic cholecystectomies have the Critical View of Safety formally confirmed before the exact step that prevents most bile duct injuries. If human surgeons, armed with full situational memory and years of training, still miss things under pressure, an autonomous robot with zero memory isn't occasionally flying blind. It's structurally, permanently flying blind — by design, every single time, unless something is built to stop it.

So we built that something: a memory agent that sits underneath the robot and refuses to let it forget. It remembers this specific patient the way a good clinician would. It remembers what's worked across thousands of similar patients the way a good institution would. And it decides — deterministically, never by vibes — when the answer is proceed, wait, or stop.

What it does

HELPIT sits between the dentist, the patient record, and the robot. Before any robotic dental procedure begins, it runs a full memory-and-safety pipeline and hands the dentist a cited, evidence-backed brief — never a robot's own unsupervised judgment call. Once the procedure starts, it doesn't walk away: a live memory co-pilot keeps consulting that same memory in real time, the instant anything drifts off plan.

Capability What it looks like Where it happens
Episodic memory Full history for this patient — prior procedures, complications, healing trajectory, reliability-weighted by record type patient_memory_server, episodic_retriever.py
Semantic memory Anonymized population patterns for this patient's bone-density class and procedure type ("D4 bone + this implant brand → 30% higher rejection rate") knowledge_base_server, semantic_retriever.py
Timely forgetting Every record gets a decay score (reliability × exp(-age/half-life), half-life set per record type); stale records are archived out of active retrieval — never deleted — and consolidated into an AI-written summary once enough pile up memory_decay.py
Cross-patient case recall Every completed case is embedded once (text-embedding-v3); a new session is ranked by cosine similarity against every other patient's history — not a per-patient scan, a practice-wide semantic index case_recall.py
The HELPIT Gate A deterministic PROCEED / PAUSE / ESCALATE decision — one hard contraindication (a material allergy, an anticoagulant flag, a scan contradiction) always outranks an otherwise-good composite score gate_evaluator.py, risk_engine_server
Live anatomy drift detection Today's scan is diffed against the patient's baseline automatically, every session, to catch bone loss or recession the existing plan doesn't account for imaging_server, vision_analyzer.py
Pre-procedure brief A cited, evidence-backed surgery-ready brief — risk flags, recommended implant angle, allergy alerts, prior complication notes — that the dentist approves or overrides brief_generator.py (Qwen3.7-Max)
Real scanned 3D previews A rotatable, real scanned tooth model per FDI position, plus a full maxillary arch reference, rendered with PBR materials and environment lighting — not procedural geometry Tooth3DViewer.jsx, DentalArchViewer.jsx
Narrated AI procedure preview One distinct Wan2.6 clip generated per planned step, captioned "STEP N," narrated by CosyVoice, stitched into a single short — genuinely generated footage per step, not one clip looped video_generator.py (Wan2.6 t2v)
Live step monitor Planned vs. actual torque/angle/depth tracked live over WebSocket; deviation past threshold pauses the procedure — and a mis-keyed value can be corrected after the fact without losing the audit trail procedure_tracker_server, LiveMonitor.jsx
Live memory co-pilot The instant a step deviates, HELPIT embeds the deviation, ranks it against every resolved past deviation in the practice, and speaks back one directive sentence grounded in what actually happened last time — or honestly says nothing similar is on file copilot_advisor.py (Qwen3.6-Flash + CosyVoice)
Post-op video analysis Upload a recording of the actual robotic session; frames are sampled across the whole clip and reviewed start-to-finish, flagging anything worth a second look — then ask it follow-up questions out loud video_analyzer.py (Qwen-VL)
Population insights An animated chart of 5-year success rate by bone-density class, plus status-colored callouts for risk patterns that don't share a common unit PopulationInsights.jsx, BoneDensityChart.jsx
Audit trail Every memory read, every gate decision, every dentist override — timestamped and citable, for exactly the malpractice-defense reason a real deployment would need it GateDecision model, evidence citations on every brief

Every gate decision ships with the specific DentalRecord IDs it was grounded in. The dentist always has the final word — HELPIT can escalate loudly, but it never overrides a human.

How we built it

Backend. FastAPI, async throughout, orchestrated by a single memory_agent.py entry point (run_gate_evaluation(session_id)) that pulls episodic + semantic memory, scores anatomy drift, evaluates the gate, generates the brief, and persists everything — with the two independent Qwen calls (anatomy narrative, surgery brief) dispatched concurrently via asyncio.gather to keep end-to-end evaluation under 15 seconds.

Memory layer. Seven FastMCP tool servers (patient_memory, imaging, procedure_tracker, knowledge_base, risk_engine, dentist_profile, outcome_monitor), dispatched in-process for latency in the hot path — but two of them (risk_engine_server, patient_memory_server) also carry a real mcp.run(transport="stdio") entrypoint, so they're genuine standalone MCP servers any MCP-compatible client can connect to over the actual protocol, not just decorated Python functions. scripts/mcp_client_demo.py proves this end-to-end: a real initialize() handshake, list_tools(), and call_tool() against the same live database the running app uses.

Model routing, matched to task, not one-size-fits-all. qwen3.7-max for the surgery brief (the highest-stakes generation call), qwen3.7-plus for grounded video Q&A, qwen3.6-flash for both the anatomy drift narrative and the live co-pilot's advisory (fast, cheap, tightly token-capped — a surgeon mid-procedure needs one sentence, not a report), qwen-vl-plus for every call that actually carries images or video frames, text-embedding-v3 for cross-patient and cross-deviation semantic recall, wan2.6-t2v for AI procedure preview video, and cosyvoice-v3-plus to speak the live co-pilot and video-analysis Q&A answers out loud.

3D anatomy, for real. Instead of hand-built primitive geometry, HELPIT ships two real scanned dental models — a single tooth (Thingiverse, CC BY-NC) and a full maxillary arch (NIH 3D, public domain, 951,316 vertices / 1.9M triangles at 34MB straight from the source). Both load via STLLoader, auto-center/rescale/orient in the browser, and share one baked PMREM environment texture across every canvas on a page.

Frontend. React 18 + Vite + Tailwind + Three.js, Framer Motion for the console's animated nav, page transitions, and scroll-reveals, and a hand-rolled SVG bar chart rather than pulling in a charting library for one chart.

Ops. GitHub Actions deploys to Alibaba Cloud ECS on every push to main, gated by a required pytest job — 42 tests covering the risk-scoring rules, the gate's PROCEED/PAUSE/ESCALATE priority logic, memory decay and consolidation, cross-patient case recall, and the live co-pilot's deviation-grounding logic, including a regression test for a real bug we shipped and fixed (below). The live deployment is reachable at a real hostname via free wildcard DNS rather than a bare IP.

Tech Stack

AI / Models (Alibaba Cloud Model Studio, via DashScope) qwen3.7-max · qwen3.7-plus · qwen3.6-flash · qwen-vl-plus · wan2.6-t2v · text-embedding-v3 · cosyvoice-v3-plus

Backend Python · FastAPI (async) · SQLAlchemy (async ORM) + aiosqlite · Pydantic v2 / pydantic-settings v2 · FastMCP (mcp Python SDK, real stdio-transport servers) · Uvicorn · httpx (async) · tenacity (retry/backoff) · native WebSockets · ffmpeg (frame extraction, caption burn-in, audio muxing, concat)

Frontend React 18 · Vite · React Router · Tailwind CSS · Framer Motion · Three.js (STLLoader, PBR materials, PMREM environment lighting) · hand-rolled SVG data visualization · Web Speech API (voice input)

Cloud & Infrastructure (Alibaba Cloud) ECS (deployment) · OSS via oss2 (object storage) · BSS OpenAPI SDK (billing) · ECS OpenAPI SDK (instance lifecycle) · Docker Compose · nip.io (free wildcard DNS)

Testing & Tooling pytest (42-test suite, CI-gating) · Playwright (browser-driven UI verification) · GitHub Actions (CI/CD)

Data SQLite (current) · PostgreSQL (planned migration path — see What's Next)

Architectural Diagram

Helpit System Architecture

Challenges we ran into

A silent data-loss bug in the step tracker. update_step fetched a session's steps (a JSON column holding a list of dicts), mutated the nested dict in place, and reassigned it to the same Python list object. SQLAlchemy only flags a plain JSON column dirty when the attribute is set to a genuinely different object — so the write silently never reached the database. The API response looked completely correct (it echoed the in-memory result), but a fresh read afterward still showed the step pending. We caught this by testing "did the change persist across a second, independent request" rather than just trusting the first response — and now there's a permanent regression test for it.

Qwen3.x's default reasoning mode blew our latency budget by 5x. Switching the brief generator to qwen3.7-max took gate evaluation from ~9 seconds to ~50 seconds with zero accuracy benefit — the model's extended chain-of-thought pass is built for open-ended reasoning, but our gate decision is deterministic Python, so there was nothing for extended thinking to actually improve. Setting enable_thinking: False brought it straight back down, verified with direct before/after timing against the live API.

Sourcing real 3D anatomy was harder than building it. Our first three attempts at procedural tooth geometry (assembled boxes/spheres, a smooth lathe-revolve) all read as "blocky toy" or "light bulb," not a tooth — a fully round, radially-symmetric shape can't represent a tooth's flat mesial/distal/buccal/lingual walls. The real fix wasn't more clever code, it was a real scanned asset. But the asset that landed — a full maxillary arch — was 951k vertices at 34MB, far too heavy to auto-rotate smoothly next to several other live 3D canvases. Three.js's standard edge-collapse simplifier was still running after 20+ minutes. We wrote a grid-based vertex-clustering decimator instead — snap every vertex to a 3D grid cell, average everything that lands in the same cell, remap triangle indices, drop degenerate triangles — an O(n) single pass that took the mesh from 951k to ~129k vertices (13.5%) in a few seconds.

CosyVoice's hardcoded 5-second handshake fought us twice. The SDK's WebSocket timeout is fixed at 5 seconds and intermittently missed it under load — even on sequential, non-concurrent calls — so the live co-pilot's narration needed both a retry policy and a switch from concurrent to sequential synthesis to stop stampeding the connection.

Windows path handling broke our own test automation. Screenshots captured via a headless browser kept silently landing in the wrong place — forward-slash MSYS-style paths get translated by Git Bash but are interpreted literally by a native Windows Node.js process, so a script that "succeeded" was actually writing files to a nonexistent directory. The fix was consistent Windows-style paths in any script actually executed by Node, not by the shell wrapping it.

Accomplishments that we're proud of

  • 100% gate accuracy and 0% false-PROCEED rate on our labeled eval harness — the single most dangerous failure mode (a false PROCEED on a patient who should PAUSE or ESCALATE) is checked explicitly and hard-fails the eval run if it's ever non-zero.
  • A live memory co-pilot that's honest, not just helpful. When it has real precedent, it gives one directive, grounded sentence — not a hedge. When it doesn't, it says so plainly instead of fabricating confidence. Verified live: a seeded depth deviation with two contrasting past outcomes produced a specific, correct recommendation; an unseeded deviation type correctly reported no precedent on file.
  • MCP integration that's actually real, not just an organizational decorator — two servers run as genuine standalone stdio MCP servers, proven with an end-to-end client script against the live database.
  • A real algorithmic contribution, not just API plumbing: the grid-based vertex-clustering decimator that took a 34MB/951k-vertex scan down to a smooth, real-time-renderable model in seconds instead of the 20+ minutes a standard simplifier needed.
  • A 42-test pytest suite covering the safety-critical gate logic, memory decay, cross-patient recall, and the live co-pilot — run as a required CI job that gates every deploy, including a permanent regression test for a real, subtle SQLAlchemy bug that shipped and got fixed.
  • Sub-15-second full gate evaluation despite routing through four different Qwen/Wan models per session.
  • An architecture we can defend as generalizable, not just dental-specific: the episodic-memory + semantic-memory + deterministic-gate + live-co-pilot + audit-trail pattern maps cleanly onto industrial robotics, other clinical decision support, and any other high-stakes automated procedure — porting it means swapping the domain models and the risk rules, not rewriting the orchestration.

What we learned

  • SQLAlchemy's dirty-tracking on JSON columns is stricter than it looks — mutate-in-place-then-reassign-same-reference is a silent no-op, and the only way to catch it is to test persistence across an independent read, not just trust a response body.
  • A bigger model isn't automatically the right model — the flagship model made our safety-critical path 5x slower for zero benefit; matching model tier to what a specific call actually needs mattered more than always reaching for the biggest one.
  • MCP's value is in the protocol, not the decorator — it's easy to use @mcp.tool() purely as code organization and never actually run a compliant server; making at least part of the system genuinely protocol-reachable is what turns "we used MCP" into something a judge (or another agent) can actually verify.
  • Memory only matters if it's consulted at the right moment — a pre-op brief is necessary but not sufficient; the moment a surgeon most needs institutional memory is mid-procedure, under pressure, which is exactly when a document is least useful and a spoken, evidence-grounded answer is most.
  • The most convincing safety story is a deterministic one — Qwen writes narrative, but the PROCEED/PAUSE/ESCALATE decision itself is plain Python with hard-coded priority ordering, which means it's testable, reproducible, and impossible for a model hallucination to quietly override.

What's next for HELPIT

  • Real EHR integration — replace synthetic patient data with a FHIR-compliant live record pull, the same direction most clinical-decision-support tools eventually need to go.
  • More distinct 3D tooth models per FDI position — right now one real scanned tooth is oriented/reused across positions; sourcing a few more real scans would let each position show its actual anatomical shape.
  • Closing the loop on the live co-pilot at scale — today it learns from deviations logged within HELPIT itself; the next step is ingesting historical deviation/outcome pairs from a real practice's existing records on day one, instead of starting from zero.
  • Postgres migration — the data layer is already async SQLAlchemy, so moving off SQLite for real concurrent load is a config change, not a rewrite.
  • Proving out the generalizable pattern for real — building a second, non-dental reference implementation (industrial robotics or another clinical domain) on the same episodic + semantic + gate + live-co-pilot architecture, to make the "this isn't just a dental app" claim concrete instead of just argued.

Built With

  • aiosqlite
  • alibabacloudecs
  • alibabacloudoss
  • docker
  • fastapi
  • fastmcp
  • githubactions
  • pydantic
  • pytest
  • qwen-vl-plus
  • qwen3.6-flash
  • qwen3.7-max
  • qwen3.7-plus
  • react
  • sqlalchemy
  • structlog
  • tailwindcss
  • tenacity
  • three.js
  • vite
  • wan2.6-t2v
Share this project:

Updates