Inspiration
Fossil research is full of detail that almost never leaves the lab. Scientists CT-scan specimens, name every bone, and publish the meshes. Most of us only ever see a flat figure in a paper.
Open repositories and museum collections already hold those prepared parts, but opening them usually means specialist software, local installs, and a lot of guesswork about what is missing or unlabeled. Studying with a paper figure does not let you turn the skull, isolate one bone, or look behind an overlap.
We built Marrow so that exploration can happen in a browser: open a curated specimen from a published study, inspect its named structures in 3D, leave a note for yourself, and keep every bone tied to the real accession and citation it came from. No account. No upload required. No AI inventing anatomy.
The collection spans multiple museum accessions and open datasets, each with its own credits and reuse terms. If a piece is missing from the source data, we say so.
What it does
Marrow is a browser workspace for exploring prepared fossil specimens in 3D.
- Explore a specimen. Browse the catalog, open a case, and see its name, museum accession, citation, and researcher credits. Rotate the skull, select individual bones, hide or focus them, and tell left from right when it matters. Switch between solid and X-ray styles, explode the skeleton to look behind overlaps, and reset the view.
- Leave a note. Pin anonymous observations on a bone. Pins stay on the same anatomical spot after you reassemble, reset, or refresh - saved in your browser only, not on a server.
- Optional helpers. Ask grounded questions about the specimen, take an AI-guided bone tour, or turn on webcam hand-control.
How we built it
- Frontend. React 19 + TypeScript + Vite, React Router, React Three Fiber / Three.js / Drei for the scene, and Zustand for viewer state. UI clicks, keyboard shortcuts, tour steps, and hand gestures all map into the same actions, so they change the view the same way.
- Backend. FastAPI (Python) serves specimen manifests, GLB meshes, health, and Gemini routes. Secrets stay server-side.
- Evidence pipeline. STL or PLY meshes to inventory with stable structure ids, convert to GLB, then manifest.json plus assets outside git, served under /specimens/. We serve prepared specimens instead of running heavy processing in the visitor path.
- Pins. Each pin stores a bone id plus bone-space (x, y, z). That spot is independent of camera and explode offset, so the note stays on the anatomy when the skeleton separates or reassembles.
- AI. Gemini through FastAPI. The model sees numbered curated facts plus manifest fields; tour steps are limited to loaded structure ids.
- Hands. MediaPipe HandLandmarker runs in the browser as an accessibility minded alternate control path. Point, pinch, open hand, and fist map into the same Zustand actions as clicks and shortcuts, so the experience stays consistent no matter the input.
Challenges we ran into
- Honest meshes. Getting real fossil meshes into the browser without inventing missing bones or blurring left and right.
- Pins that stay put. A note has to stay on the same anatomical spot after explode, reassemble, and reset - not just stuck to a screen click.
- AI as a helper, not a source. Answers had to stay grounded in curated facts instead of inventing labels or measurements.
- Scope under time pressure. Keeping four parallel tracks (API, 3D scene, UI shell, and pins/attribution) aligned across a growing specimen catalog.
Accomplishments that we're proud of
- A visitor can open a real museum specimen, inspect named bones, look behind overlaps, and pin a note that survives explode and refresh.
- Left and right structures stay distinct from data through highlight and caption.
- Hand control is a real second way into the viewer: the same select / explode / reassemble path works without relying on fine mouse control, and it stays optional so it never blocks the core demo.
- Accession, citation, source DOI, and researcher credits stay visible in the product across the catalog.
What we learned
- Serving prepared, attributed specimens works well for fossils - the same pattern that works for clinical imaging demos.
- Stable structure ids matter more than display names when selection, hide, focus, and tours all have to stay correct.
- Splitting evidence prep, API, and the React shell made parallel hackathon work actually parallel.
- Calling "X-ray" an opacity style (not a real radiograph) kept us honest with judges and visitors.
What's next for Marrow
- Real CT slice viewing linked to meshes only when alignment is verified - never fake slices.
- Richer curated facts per specimen so grounded Ask covers more of each published study.
- Grow the catalog with more open museum accessions.
- Shared notes or rooms later, only after the local anonymous pin story stays clear.
