Inspiration
A brain MRI shows anatomy one slice at a time. To understand how those slices fit together, a student has to build a 3D picture in their head. Studying with someone else adds another problem: making sure you're both looking at the same place.
We built Lobe to make that exploration visual and shared: open a scan, see its structure in 3D, talk it through together, and practice on the same scans afterward.
What it does
Lobe is a collaborative workspace for studying brain MRI scans in 3D.
Explore a case. Pick one of 27 prepared cases from the library or upload four MRI sequences (T1c, T1, T2, FLAIR). Study the scan in three linked slice views with the tumor overlay, then open the same case as an interactive 3D brain. Rotate it, make it see-through, pull the two hemispheres apart, hide a hemisphere, and toggle the tumor layers (edema, core, enhancing). A case panel shows tumor volumes, side, lesion count and the model's certainty.
Review it together. Share an invite link or QR code. Guests follow the presenter's page, slices, camera and display settings, can break away to look around on their own, and rejoin with one click. Named pointers show where each person is pointing on the brain. The room has chat, threaded notes pinned to the scan, a 3D whiteboard (sticky notes, text, pen) and a voice call with a live transcript.
Ask the assistant. A Claude-powered assistant answers questions about the case and can move the viewer for you: zoom, rotate, isolate a region, switch modes, take a screenshot. A second agent takes meeting notes. When the session ends, each member who left an email gets a Word report with the findings, notes, chat, transcript summary and screenshots. The assistant refuses requests for medical advice.
Practice. Four quick games built on the same scans: Find the Slice, Beat the AI (outline the tumor and get scored against the expert outline), Which Side?, and Bigger or Smaller. Play solo or start a playground for your study group, with a host and a leaderboard.
Use it from Claude. An MCP server lets Claude Desktop or Claude Code read case reports, anatomy, chat and notes, and export a session.
Tumor outlines come from segmentation, which labels regions in an image. Uploaded scans go through a pretrained model; prepared cases load results we generated ahead of time.
Lobe is an educational hackathon prototype. It is not clinically validated and does not give diagnoses or treatment recommendations.
Architecture
The React app talks to the imaging service for scans and meshes, and to Convex for everything shared. Convex calls out to LiveKit, Anthropic and AgentMail, and exposes an MCP endpoint for Claude Desktop and Claude Code.
How we built it
Frontend. React, React Three Fiber / Three.js for the 3D brain, and NiiVue for the slice views. Every input (mouse clicks, keyboard shortcuts, the assistant's tool calls) goes through one typed action dispatcher, so they all change the view the same way.
Imaging service. A Python FastAPI service in Docker validates the uploaded scans, runs MONAI's pretrained SegResNet model, and turns the masks into 3D surfaces. Brain and tumor meshes use the same scan transform, so they share one coordinate system. We used an existing model rather than training our own. In production it runs on RunPod.
Collaboration. Convex stores sessions, presence, pointers, chat, notes, whiteboard items, game rounds and transcripts, and its live queries keep everyone in sync. It also hosts the built frontend. LiveKit carries the voice call. Speech-to-text runs in each browser, so no audio is sent to our servers.
AI. The assistant runs on Anthropic's Claude (Opus for answers, Haiku for meeting notes). It controls the viewer through a fixed set of typed tools and reads case data through the same MCP tools that Claude Desktop uses. AgentMail sends the session report.
Pointers. A named pointer stores a spot on the model's surface. Each person's screen maps that spot into their own view, so the pointer stays on the same anatomy even when their display settings differ.
CI/CD. GitHub Actions runs the Convex, frontend and backend test suites on every push. On main it deploys Convex plus the frontend, builds the imaging image, and rolls the RunPod pod.
Challenges we ran into
Coordinates. MRI volumes, 3D surfaces and shared pointers each need explicit coordinate handling. A small conversion bug looks like "the tumor is 4 mm to the left," which is easy to miss. A pointer also has to move with the brain when the hemispheres separate.
Following without trapping. Following a presenter should help learners without stopping them from exploring. We built an explicit follow, break away and rejoin flow to support both.
An AI that can act safely. The assistant can change what everyone sees, so it only gets a fixed set of typed tools, checks session membership, has a daily run limit, and refuses medical-judgment questions.
Scope. We had to separate what works from what's unfinished. See-through mode and hemisphere separation work; a true cutaway view is still future work.
Accomplishments that we're proud of
We put scan exploration, live discussion and practice into one workflow, from upload to emailed report.
In browser testing, two participants opened the same case, followed the presenter, explored on their own, rejoined, saw each other's named pointers stay on the anatomy at different explode settings, and exchanged messages. That shared interaction is the core of Lobe.
What we learned
Detailed graphics alone don't make complex information understandable. Clear controls, an easy way to get back to your view, and a shared reference point matter just as much.
We also learned to separate visual alignment from medical correctness. An outline drawn in the right coordinate system can still have the wrong boundaries.
What's next for Lobe
- Test the voice call, the meeting notes and the session report end to end on two devices.
- Finish scan-coordinate handling for annotations, and add a true cutaway view.
- Add hand-gesture and voice control.
- Improve accessibility, and get feedback from students and educators before adding more features.
Built with
React · TypeScript · Vite · Three.js · React Three Fiber · NiiVue · Zustand · Convex · LiveKit · Anthropic Claude · Model Context Protocol · AgentMail · Python · FastAPI · PyTorch · MONAI · nibabel · scikit-image · trimesh · Docker · RunPod · GitHub Actions
