Rainbone

See the break before it happens. From CT scan to 3D stress map to prevention plan explained by AI agents.

Comment

Full demo video (2:39): https://youtu.be/SCrmSu9aVzs · Highlights (1:59): https://youtu.be/81SZ7VXBaV0

Rainbone in 30 seconds

Rainbone combines CT-derived 3D models, physics simulations and checked AI to help the care team see bone weakness early, before a fall becomes an emergency.

Rainbone starts from real research CT scans of the thighbone, each with a recorded follow-up. Rainbone shows, in 3D and in colour, where a real femur takes the most strain under load. Then Gemini explains in plain words, for the care team, how that femur compares with the published research cohort. Try it: https://rainbone.health

Built for PointClickCare's Catch It Early challenge: catch bone weakness early, so prevention can start before the fall.

Who it's for: the care team in nursing homes and senior living: nurses, CNAs, wellness directors, doctors and admin staff.

Under the hood: an AI agent pipeline (Ask Rainbone) orchestrated with LangGraph on the Gemini API; finite-element physics on Vultr Compute, with an optional C++23 mesh kernel; sha256-pinned facts in Vultr Object Storage; run history and an AI audit trail in Tiger Data's TimescaleDB; all on a GoDaddy Registry .health domain.

A research prototype built on public, de-identified research data. Not a medical device.

Videos: the full walkthrough (2:39) is above; a 1:59 highlights cut is at https://youtu.be/81SZ7VXBaV0

Try it in 60 seconds

  1. Open https://rainbone.health and press Enter the lab. The demo seat (Dr. Rowan, care team) needs no account.
  2. Patient overview: rotate the femur and turn on Cut-away to see the stress inside the bone.
  3. Physics simulations: scrub the press, press Go to first break, and download the missing region as an STL to 3D-print it at 1:1 scale.
  4. AI cards: read the provenance line under each card: which model wrote it, and that its facts were verified from Vultr Object Storage.
  5. Ask Rainbone: click What does the 47% mean? and watch the agents run: triage, evidence, writer, critic, audit.
  6. Secure datasets → Recent AI answers: the audit rows Tiger Cloud stored for those answers.

Also on the site: a research paper with the full method and the maths (https://rainbone.health/research-paper: 69 numbered equations, the LangGraph state machine, a French résumé), a printable simulation report on Physics simulations, and a smoothed 3D lesion view (display smoothing only; the lesion progression is illustrative).

Inspiration

A bone can look fine on a grey 2-D CT slice and still be close to breaking. Clinicians get pictures and checklists. The people looking after an older woman rarely get a plain answer to "how strong is her bone?" until a fall gives them one. We wanted to turn a CT scan into something you can see, rotate, cut open and explain: where a bone works hardest under load, and what that means in plain language.

Then we found HFValid, a dataset from the University of Bologna. It covers 101 women aged 56 to 86, with a femur CT for each woman, the study authors' own fall-fracture model results and a record of who fractured over the following five years. That gave us a real bone, real numbers, and someone to explain them to: the care team in senior living, where a fall is the moment a weak bone becomes an emergency.

What it does

The lab opens on one real case: HFValid subject Pat001, the complete right femur of a woman who was 78 at her scan.

  • A real femur, solved. The femur is meshed into 60,819 tetrahedra and 17,361 nodes and solved with SfePy, with 800 N on the femoral head and the distal 15 mm fixed. The 3D viewer colours the bone by von Mises stress (0 to 15.83 MPa, nodal-averaged) or by displacement. It renders live in your browser with vtk.js.
  • Cut it open. The cut-away slices the solved tetrahedral mesh in the browser, so you read the stress inside the bone and not just on its surface. The cut stays facing you as you rotate, and you can zoom into the cut face.
  • Where she stands in the research cohort. Next to our solve, the Clinical metrics card shows the study authors' published numbers for her femur:
    • Her absolute risk of fracture at time 0 (ARF0) is 47.1 % computed by the AI, higher than 56 of the 101 women. The study median is 44.3 %, and the 4 women who fractured were at 53.7 to 73.2 %.
    • Her minimum sideways-fall strength (MSF) is 1,339 N, weaker than 75 of the 101.
    • A sideways fall hits the hip with 2.90 to 4.26 kN (van den Kroonenberg 1995), which is 2.17 to 3.18 times her femur's strength.
    • The study recorded no fracture for her during the follow-up.

Our own standing-load stress ratio sits beside these numbers: 15.8 MPa against an assumed 150 MPa bone strength, or 10.6 %. It is labelled as what it is, an engineering ratio and not a probability. A What do these numbers mean? panel tags every number on the card as published data, computed by Rainbone, derived or assumed.

  • AI that explains in plain words. There are two cards:
    • AI fracture risk analysis: Gemini puts the published facts into two or three sentences for a care team.
    • In plain words: Gemini explains the femur's own solve (load, stress, stiffness and bone volume) by comparing the bone to a bridge.

Every number in either text is checked against the facts before it's shown. Each card names the model that wrote it and says where its facts were read from: Vultr Object Storage with a verified sha256, or the bundled backup copy and why.

  • A virtual press test. Physics simulations replays the same femur in a virtual hydraulic press over 15 solved frames. The force peaks at 20,079 N, then falls as elements pass an assumed strain limit and modelled damage spreads. The page labels it as exploratory.
  • 3D-printable surgical references & joint replacements. You can download the full bone as a print-ready STL at true scale, or download the exact missing region of a fracture. This provides the precise volumetric dimensions researchers and biomedical engineers need to design and 3D-print custom, porous "spongy" joint replacements, bridging the gap between digital simulation and physical treatment.
  • A live engine. Each visit sends a small synthetic CT volume to our FastAPI back end on Vultr. The back end keeps voxels at or above 150 HU, maps CT values to stiffness, meshes the bone and solves an 800 N load with SfePy. Each new solve is saved to Tiger Cloud, the dashboard reads the run back, and an hourly continuous aggregate summarises the runs. Import scan runs the same solve on your own .nii or .nii.gz CT.
  • Imaging library. It holds three CTPelvic1K pelvic CT scans with six bones each: sacrum, both hip bones, both femurs and the L5 vertebra. TotalSegmentator segmented them, and the meshes are kept in a private Vultr Object Storage bucket. The browser loads each mesh through a 15-minute pre-signed link and checks its sha256 before drawing it.
  • Also in the lab:
    • Secure datasets states where each dataset lives and who can read it.
    • Preferences offers light, dark and high-contrast themes, large text and reduced motion.
    • Lesion progression is a visualization demo based off of the Longitude CT dataset.

How we built it

  • Front end: Next.js 15, React 19, TypeScript and Tailwind CSS on Vercel. vtk.js draws every bone: the stress map, the cut-away, the press frames and the pelvic meshes. Vercel forwards /api/* to the back end, so API calls stay on rainbone.health. Only the pelvic meshes come straight from the Vultr bucket, through signed links.
  • Back end: FastAPI on uvicorn behind Caddy HTTPS, on a Vultr Compute VM at api.rainbone.health. It runs one solve at a time and answers a file it has already solved from memory.
  • Femur pipeline (offline):
    1. HFValid CT and the authors' segmentation go into 3D Slicer 5.12.4, which exports a closed surface.
    2. TetGen 1.6.0 turns the surface into tetrahedra.
    3. SfePy runs a linear elastic solve with one assumed material (E = 10 GPa, ν = 0.3).
    4. The results become VTU and VTP files served with the site.

The support reaction balances the 800 N load to better than one part in a billion.

  • Imaging pipeline: CTPelvic1K scans go through TotalSegmentator 2.18.0, become VTK surface meshes, and are uploaded to private Vultr Object Storage with boto3. Each object's sha256 is stored in its metadata and in a manifest.
  • Data: Tiger Data (Tiger Cloud, TimescaleDB) holds two hypertables:
    • simulation_runs, with a real-time hourly continuous aggregate
    • ai_analyses, which audits every AI answer
  • AI: the Gemini API (gemini-3.8-flash, then gemini-3.5-flash-lite), Groq (openai/gpt-oss-120b) and a deterministic template. Both providers return structured JSON.
  • Domain: rainbone.health. The .health extension is run by GoDaddy Registry, and we registered the name through Porkbun. The apex serves the Vercel site, and api. points at the Vultr VM.
  • Optional C++23 kernel. Mesh boundary-face extraction runs in an optional C++23 kernel (C++20, then C++17 fallback) that measured 15.5–16.1× faster than the NumPy path on the production API host (6.3× locally) on the 60,819-tetrahedron femur. If it can't be built, the NumPy path runs instead. It speeds up that one stage, not the whole solve.

The AI layer

Our rule: code computes, and the model only puts the results into words.

  1. Facts first. The API builds every number a model may use: Pat001's ARF0, her two ranks, the cohort median and fracture range, and the fall-force ratios. Models receive only those numbers, never a scan or a file name, and are told to calculate nothing.
  2. Grounded in Vultr. The API reads the facts from a private clinical/ folder in Vultr Object Storage. The index's sha256 must equal the hash pinned in our repository, and the object's bytes must hash to that value. If the bucket is slow (1 s to connect, 1.5 s to read), unreachable or doesn't match, the API answers from the copy bundled with it and says why. A changed bucket can never quietly change the facts.
  3. A checker on every answer. Before any text is shown, code rejects it if it does any of these:
    • uses a number that isn't a fact
    • uses a fact as the wrong quantity, such as an age as a percent or a rank as a risk
    • states a rank the wrong way round
    • writes a number as a word, unless it is a count of women that matches a fact
    • says when or whether a fracture will happen
    • calls ARF0 a chance
    • gives advice or names a disease
    • puts the recorded fracture at the hip (the data records that a fracture happened, not where)
  4. Fallbacks, honestly labelled. The chain runs in this order:
    1. gemini-3.8-flash
    2. gemini-3.5-flash-lite
    3. the Gemini answer saved earlier for exactly the same facts, checked again before it's shown
    4. Groq's openai/gpt-oss-120b
    5. a deterministic template that always passes the checker

When gemini-3.8-flash is rate-limited (HTTP 429), the API remembers that for 300 s and the lighter gemini-3.5-flash-lite answers instead. Each card says where its text came from: Gemini · gemini-3.5-flash-lite, Gemini · saved answer from a given time, Backup model · Groq or Offline summary (no AI). Gemini's answers are cached for an hour and the others for a minute, so Gemini gets asked again soon. Page loads that arrive together share one model call.

  1. Audited in Tiger. Every answer a model or the template produces writes one ai_analyses row. The row records the endpoint, case, provider, model, the fallbacks taken and why, the latency, where the facts were read from, and the sha256 of the facts and of the text. It never stores the text itself. A saved answer isn't logged a second time, because no model was called for it. Anyone can read the latest rows at https://rainbone.health/api/ai-log.

It works in production: our live log shows the checker catching an answer that stated a rank the wrong way round and serving a checked answer instead.

Ask Rainbone: AI agents you can watch. A chat drawer answers questions about the case, the pages and the data. The drawer shows each step as it runs: triage (by rules, or Gemini's native function calling picks the tools), read-only tools (patient facts, cohort comparison, FEA result, page guide, dataset info), a writer that may only pick statements from a fixed, checked catalogue, a code critic that can ask for one rewrite, and an audit row in Tiger. The five stages (triage, evidence, writer, critic, audit) run as a LangGraph StateGraph, with a conditional edge from the critic back to the writer for that one rewrite. If the framework itself fails, a built-in runner carries on from the same stage without re-running finished ones. No checkpoints and no external tracing; the drawer's trace names the orchestrator that ran. A token meter under each answer shows the tokens it used, with Google's published median energy for a Gemini text prompt for scale (not measured for that answer).

Challenges we ran into

  • The Pivot: Our very first idea was a smart scrunchie, but it required an accelerometer we couldn't get our hands on. We completely pivoted our project on Saturday afternoon.
  • Data Bottlenecks & Lag: Converting from 2D CT scans to the .vtp format generated massive files that caused severe browser lag. We solved this by migrating our storage to Vultr Object Storage (S3-compatible) and using Vultr Compute instances to handle the heavy backend solvers. This allowed us to stream the 3D meshes instantly without crashing the client.
  • Building the inside view: Showing the inside of a bone meant building the cut-away tool entirely from scratch, because the base 3D library couldn't read the solved volume mesh natively.
  • AI Math Hallucinations: Some physics in our initial AI prompt was wrong (e.g., solid-bone stiffness was being used for spongy bone). We cross-referenced everything with medical literature, fixed the math, and hard-coded the AI to prevent it from ever guessing numbers.
  • Design & Trust: Red on a stress map doesn't necessarily mean "about to break." We built a separate failure-proximity view that records loading history against an assumed strain limit. We also focused heavily on a clean, clinical UI (bone-white and marrow-navy) so the platform felt trustworthy, keeping bright colors strictly reserved for actual data representation.

Accomplishments that we're proud of

  • A real femur, solved end to end. A complete femur from CT, 60,819 tetrahedra solved with SfePy and rendered live in the browser, cut-away included.
  • AI agents you can watch. Ask Rainbone runs a five-stage agent pipeline orchestrated with LangGraph, with Gemini function calling at triage, and shows every step.
  • AI that is useful and honest. It never invents a number: every number it writes is checked, every answer is logged in Tiger Cloud, and each card names the model that wrote it.
  • Red-teamed our own AI. 32 hostile questions to the live Ask Rainbone, scored by fixed rules: 30 passed, and the two misses are fixed and pass live. Report: https://github.com/galitau/Rainbone_prod_hackthevalley11/blob/main/redteam/REDTEAM.md
  • Print the simulated break. Download the missing region of the simulated break as an STL and 3D-print it at 1:1 scale.
  • Fully deployed. Live at https://rainbone.health on Vercel, Vultr Compute, Vultr Object Storage, Tiger Cloud and a .health domain.
  • Segmentation that holds up. Dice 0.91 to 0.94 against the dataset's expert masks for the sacrum and both hip bones, after we merged TotalSegmentator's separate S1 label back into the sacrum.
  • Native speed where it counts. An optional C++23 kernel makes mesh boundary extraction 15.5–16.1× faster on the server, with a Python fallback.
  • A research paper to back it. The full method and maths at https://rainbone.health/research-paper: 69 numbered equations and the LangGraph state machine.
  • Designed for the people reading it. Light, dark and high-contrast themes, large text, reduced motion, and a smooth 3D lesion view.

What we learned

We learned how CT numbers map to bone stiffness. We also learned why a patient-specific shape doesn't make a model clinically validated: the material, the load direction and the way you summarise the stress field (nodal average or single element) all change the answer. A ratio is not a probability. A language model follows rules most of the time, and a checker is what decides what reaches the screen. Finally, a registry is not a registrar: GoDaddy Registry runs .health, and we bought the name through Porkbun.

What's next

  • Catch it where care teams already work. Once the approvals are in place, run the check alongside the care home's existing records system (such as PointClickCare), so a new CT leads to a flag in the place the team already looks. We have no PointClickCare integration today.
  • A timeline for each resident, once the right approvals are in place. With resident consent, a research-ethics review and a PHIPA-compliant agreement with the care home, each new CT would be solved and compared with her last one under a de-identified ID, so a care team sees her bone changing before a fall. Privacy comes first: consent, encryption and access control. Today Rainbone stores no patient records.
  • CT-density-calibrated material for the femur instead of one assumed stiffness, plus a mesh-convergence study.
  • Run the femur pipeline on all 101 HFValid femurs, so the card can show any of them and we can compare our solve with the recorded outcomes.
  • Reach the goal in our README: run the same check on a hip CT a resident already has, instead of reading published numbers.
  • Sit down with care teams to learn where this belongs in a real assessment, before anyone treats it as more than research.

Built with Python, FastAPI, Uvicorn, SfePy, NumPy, SciPy, TetGen, 3D Slicer, TotalSegmentator and C++23; Next.js, React, TypeScript, Tailwind CSS and vtk.js; AI agents on the Gemini API (Google AI Studio) orchestrated with LangGraph, with Groq as backup; Vultr Compute and Vultr Object Storage; Tiger Data (Tiger Cloud, TimescaleDB on PostgreSQL); Vercel, Caddy and Google Sign-In; and a .health domain from GoDaddy Registry.

Built With

Share this project:

Updates

Submission history