Virtual Trial

Every change to insulin dosing code runs a virtual clinical trial on simulated patients inside GitLab. Duo agents plan and explain it; risky changes are blocked before release.

Comment

Virtual Trial gives every change to insulin dosing code a clinical trial on virtual patients, inside GitLab, before it can merge or ship. GitLab Duo agents plan the trial, explain the result and respond to incidents. The pipeline enforces human-set safety limits. A person makes the final call.

▶ 3-minute demo · Live site · GitLab project · Every claim with its evidence

Inspiration

People with type 1 diabetes let software decide their insulin every five minutes. When that software gets it wrong, people get hurt: one pump app update led to 224 reported injuries [1], and an app error could deliver 10 to 100 times the intended dose [2].

Thousands of people build and use open-source dosing systems such as OpenAPS, AndroidAPS and Trio, which build on the same algorithm, oref0. Their checks confirm that the code gives the expected output for fixed inputs. None of them asks the question that matters: what does this change do to a person over a day?

What it does

Virtual Trial asks that question automatically on every merge request to OpenAPS oref0.

  1. A person opens a merge request. Nothing else is needed until the final approval.
  2. Agents plan it (automatic). Within seconds, GitLab's Code Review flow reviews the code, and our plan agent reads the change, names the patient-safety risks and sizes the trial: 6 virtual patients, or all 30 for risky changes. It can ask for more testing, never less.
  3. The trial runs (automatic). The pipeline runs the old and new code side by side on the same virtual patients through 10 difficult days: a forgotten meal dose, a large dinner, a basal rate set too high, and five faults modeled on real bugs and recalls (sensor dropout, a false low reading, missing pump history). A worst-day hunter then searches dozens of varied days for the one that hurts the new code most.
  4. The gate decides (automatic, no AI). If the risk of dangerous low or high blood sugar rises beyond human-set limits, the change is blocked and cannot merge. Matched old-versus-new comparisons catch small but consistent harm that averages hide.
  5. An agent explains (automatic). About a minute after a block, a two-agent Duo flow posts the reason on the merge request in plain words, pointing at the line that caused it.
  6. It ships safely (automatic). A merged change deploys to Google Cloud Run as a canary that gets no traffic until 151 dosing decisions pass. Each version becomes a GitLab Release with its evidence attached.
  7. Production is watched (automatic). Ten virtual patients live on the deployed engine around the clock. If they do worse after a release, an incident opens by itself, and a Duo agent traces it to the exact commit and opens a revert, which goes through the trial again.
  8. A person approves. Only a person can merge. No agent can write dosing code, change the trial or loosen the limits: their tools don't allow it, and a pipeline job checks every merge request.

On our demo merge request, nobody touched anything after opening it: the agents started in 7 seconds, the full trial was chosen at 63 seconds, the gate blocked the change at 23 minutes, and the explanation was posted at 24 minutes.

What we found

  • All 9 harmful changes caught, with no false alarms. On 9 harmful and 7 harmless seeded changes, Virtual Trial caught 9 of 9 and wrongly flagged 0 of 7. oref0's existing checks caught 6 of 9 and wrongly flagged 3 of 7. results
  • Averages hide harm. Raising one insulin safety cap 4x increased low-blood-sugar risk by 30%, worse in 222 of 300 matched runs, yet no average crossed its limit. Only the matched comparison caught it.
  • A real bug from 2017. We put a bug that oref0 once shipped back into today's code: it kept insulin flowing during a low. It passes oref0's own checks, and Virtual Trial blocks it. history
  • The agent's judgment changed the outcome. A lowered low-glucose threshold passes a 6-patient trial. The plan agent recognized a moved safety threshold and asked for all 30 patients, which found a severe low and blocked the change.
  • It works beyond oref0. The dosing code of Trio, an iOS app, ran through the same reusable GitLab CI component unchanged.
  • Measured carbon: 7.89 g CO2e per trial run [7]. The plan agent orders the large trial only when the risk calls for it.

How we built it

GitLab Duo Agent Platform: four custom flows with six agents (plan, explain, incident response, weekly safety digest) and a chat agent, published to the AI Catalog and started by triggers: merge request opened, pipeline failed, incident opened. GitLab's own Code Review, Fix CI/CD Pipeline, SAST False Positive Detection and Resolve SAST flows also work on the project. An MCP server on the dosing engine lets GitLab Duo look up doses, verdicts and clinic status.

GitLab CI/CD: the trial as 6 parallel jobs, the safety gate, security scans (SAST, secret detection, dependency scanning), the agent-boundary check, code owners for safety files, a canary release with automatic rollback, GitLab Releases with evidence, an hourly production watch, and a reusable CI/CD component.

Google Cloud: Cloud Run (dosing engine, website, virtual clinic), Artifact Registry, Cloud Storage, Cloud Scheduler and Cloud Monitoring, deployed without stored keys through Workload Identity Federation, all defined in Terraform.

Trial engine: Python, with oref0's JavaScript dosing virtual patients from simglucose, an open-source version of the UVA/Padova type 1 diabetes simulator [3]. Outcomes use standard clinical measures: time in range, time below 70 and 54 mg/dL [4], and the low and high blood glucose risk indices [5].

Challenges we ran into

  • Units: oref0 doses in units per hour, the simulator in units per minute. One mix-up silently overdosed every virtual patient 60x.
  • Averages hide harm: a dangerous bug barely moved the average. Matched old-versus-new runs and an exact statistical test fixed that.
  • Children: adult-size meals overdosed small virtual patients and hid real differences, so meals now scale with body weight.
  • Useful but bounded agents: giving agents real responsibility without ever letting them touch dosing code.

What we learned

Agents are most trustworthy when they cannot act on the thing that matters most. Planning, explaining and proposing a revert was enough to make them useful; the pipeline and a person do the rest. And for safety, averages are not enough: compare the same patient on the same day.

What's next

  • Replay more real bugs from the histories of AndroidAPS, Trio and Loop.
  • Offer the CI component to OpenAPS and Trio maintainers and learn which failures matter most to them.
  • Add exercise and illness to the virtual patients.

References

  1. U.S. FDA. Tandem Diabetes Care recalls version 2.7 of the t:connect mobile app (Class I recall, 224 reported injuries), 2024.
  2. U.S. FDA. Insulet recalls Omnipod 5 Android app due to software error.
  3. Dalla Man C. et al. "The UVA/PADOVA Type 1 Diabetes Simulator: New Features." J Diabetes Sci Technol, 2014. Open-source implementation: simglucose, Xie J., 2018.
  4. Battelino T. et al. "Clinical Targets for Continuous Glucose Monitoring Data Interpretation: Recommendations From the International Consensus on Time in Range." Diabetes Care, 2019.
  5. Kovatchev B. P. et al. "Symmetrization of the blood glucose measurement scale and its applications." Diabetes Care, 1997.
  6. OpenAPS oref0 (MIT) and trio-oref (MIT).
  7. Green Software Foundation. Software Carbon Intensity specification (ISO/IEC 21031:2024).

Research and education only. Not a medical device. All patients are simulated, and results compare old and new code; they do not predict outcomes for any person.

Built With

  • artifact-registry
  • astro
  • cloud-monitoring
  • cloud-scheduler
  • cloud-storage
  • docker
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • gitlab-duo-agent-platform
  • google-cloud
  • google-cloud-run
  • javascript
  • nginx
  • node.js
  • openaps
  • oref0
  • python
  • simglucose
  • terraform
  • workload-identity-federation
  • yaml
Share this project:

Updates

Submission history