Auspex: Traffic Control for Hospital AMR Fleets

Robots yield. Patients don’t wait. Auspex is the urgency-aware reasoning layer for hospital fleets: it combines every data source with NVIDIA Nemotron so the most critical move goes first.

Comment

Inspiration

At 10:40, the patient in bed 12 needs oxygen. A delivery robot is carrying it. At 10:31, a porter wheels a patient on a bariatric bed into the only lift large enough to take it. That's the lift the robot is heading for.

The hospital already knew about that transport. It was in the porter system as a Stat job, with a destination, a bed type and a note: "on portable vent, needs large car." The robots' coordination software, Open-RMF, knew nothing about it. Open-RMF is excellent at keeping robots from colliding with each other, but it only sees robots. So the robot arrives, finds the lift taken, and waits. Nobody knows whether those lost minutes matter.

I wanted to know whether this was a real problem or an imagined one, so I asked someone who builds these systems. I could get in touch with Mr. Brandon Koh, a senior robotics engineer at CHART, the team behind Singapore's hospital robotics middleware, who shared from his personal experience four things:

  • Beds near lifts routinely obstruct robots.
  • Open-RMF doesn't consider obstacles outside its own traffic system.
  • Robots report "blocked" without knowing why, and someone often has to walk over and look.
  • Open-RMF has no idea whether a delayed delivery matters to the ward waiting for it.

That gap became Auspex.

What it does

Open-RMF moves the robots. Auspex decides what each movement means for the people waiting.

Auspex sits above Open-RMF and joins two worlds that never talk: the hospital's transport data (patient transports, delivery requests with deadlines) and the live state of the autonomous mobile robot (AMR) fleet. It works in two modes.

Proactive. Auspex reads transport jobs as they change status. It forecasts which lifts and corridors each mover will need, and when, and spots conflicts before a robot gets there. Then it steers Open-RMF around them, for example by closing a lift's lanes to one fleet and asking the robot to replan mid-task.

Reactive. When a robot stalls for a reason nobody recorded, Auspex doesn't stop to diagnose. It asks what's at stake. A stalled linen run can wait. A stalled oxygen delivery with six minutes of slack gets a decision: wait, reroute, or hand the delivery to another robot. Afterwards, Auspex writes an incident report listing the possible causes, each with its evidence, and none asserted as fact.

Hospital data is messy, so the interesting cases are the ones where the answer isn't in the structured fields. That's where NVIDIA Nemotron does the work:

  • S1: interpretation. A Stat job arrives with destination "CT 2" and a note about a ventilator. No field says which floor or which lift. Nemotron works out that it's Imaging on L1, and that a bariatric bed means one specific lift. Only then does the conflict with the oxygen robot appear, and the robot switches lifts mid-task.
  • S2: uncertainty. The oxygen robot stalls at a lift reporting fault code E-17. Tavily retrieves the lift vendor's bulletin: E-17 clears itself in about two minutes, whereas E-42 needs a technician. Nemotron weighs waiting against a handover, and cites the evidence.
  • S3: a trade-off, with a person in the loop. Two Stat patients and the oxygen delivery need the same lift. Nemotron proposes combined plans. Ordering two Stat patients is a clinical decision, so Auspex sends it to a person to approve, and both outcomes (approve and reject) are handled.

Throughout, the LLM never moves a robot. Every plan is a typed object, checked by a deterministic policy gate, applied only through Open-RMF, verified against live state, and recorded in a tamper-evident, hash-chained ledger.

For judges: Try it: auspex-theta.vercel.app. The site replays the recorded runs on a 2D map of the clinic. You can compare Open-RMF alone (A), Auspex with rules only (B) and Auspex with Nemotron (C), side by side. It shows every decision with Nemotron’s reading, the options and the gate’s verdict. The audit ledger can be verified in your browser, or tampered with to watch the verification fail. The Playground lets you edit S1’s messy transport note, including trying a prompt injection, and calls Nemotron live to show how it reads the note and what Auspex decides. Code: github.com/ambica-p/auspex. A few things to note- The playground might not start immediately, because of how Render free-tier hosting works. The simulation required a gpu to run, which I did not have credits for and hence couldnt make the whole thing live on Nebius Cloud or another service.

How I built it

  • Simulation: ROS 2 Kilted, Open-RMF and Gazebo, running Open-RMF's two-floor clinic world with two robot fleets and two lifts.
  • Core: Python 3.12 and FastAPI, with Postgres for the event queue, the operational picture and the ledger. A separate bridge process is the only part that talks to ROS.
  • Steering Open-RMF without modifying it: Open-RMF exposes no way to ask a fleet to replan from outside. I wrapped the demo fleet adapter, leaving its code unchanged, and added a replan service. Lane closures, task interrupts and handovers all go through Open-RMF's own interfaces.
  • NVIDIA Nemotron on Nebius Token Factory:
    • Nemotron Nano (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B) interprets incomplete hospital data, extracts structured facts from vendor bulletins, and writes incident reports. These jobs need speed and reliable structured output.
    • Nemotron Super (nvidia/nemotron-3-super-120b-a12b) makes the decisions: weighing uncertain delays and trading off competing jobs, using read-only tools to inspect the timeline, evaluate options and check policy.
    • Both return JSON-schema-validated output. One repair retry is allowed, and if the output is still invalid, Auspex fails closed to the best deterministic option.
  • Tavily retrieves the vendor's advisories for a fault code. When search finds nothing (a newly published site isn't indexed yet), Auspex reads the vendor's bulletin index with Tavily Extract. Results are treated as untrusted data: delimited, never followed as instructions, and used only after validation. One bulletin even carries an injected "instruction" in a comment, which is ignored.
  • The design rule: arithmetic is deterministic; ambiguity is Nemotron's job. Time overlaps, arrival estimates and policy checks are computed exactly. Every Nemotron proposal is re-scored by the same deterministic evaluator before the gate sees it.
  • The hospital mock: synthetic data shaped like real transport systems (EHR transport modules, porter dispatch), deliberately built from edge cases where structured fields alone aren't enough.
  • The judge site: Next.js, deployed on Vercel. It replays runs recorded in the final rehearsal on real Nemotron and Tavily, and verifies the ledger's SHA-256 chain in the browser.
  • The Playground backend: a small FastAPI service with no ROS or database, deployed as a Docker container on Render. It runs the same interpretation prompt, evaluator and gate as the core, on a recorded S1 snapshot, and calls Nemotron Nano on Token Factory live. Its controls are a kill switch, a daily cap, a per-IP rate limit, a cache and a $0.02 spend budget. The spend and the daily count are kept in Upstash Redis, so they survive the free host restarting. When any limit is reached, judges see the recorded decision and the reason.
  • Benchmark: I ran the A/B/C benchmark locally rather than on Nebius Serverless Jobs, to fit the deadline.

Challenges I ran into

I had to rethink my core assumption. I started by assuming hospitals track patient transports in real time, which would make every conflict predictable. My practitioner contact, and my own research, showed otherwise. Transport requests are often recorded digitally, with priorities and status changes, but live positions mostly aren't, and many bed movements are never recorded at all. I redesigned around the data that actually exists: prediction where job data is available, and fast, stakes-aware response where it isn't.

The simulation fought back.

  • Out-of-memory crashes took down the whole Linux environment.
  • A shared-memory transport in ROS 2 silently hid one robot fleet from the rest of the system.
  • Small default network buffers dropped the 2 MB building map, so robots never registered.

In a real hospital, each of these would have meant a fleet that silently disappears. So the simulation now refuses to start without the right settings.

Open-RMF's behaviour had to be discovered, not assumed. I tested every action before relying on it, and some results were surprising:

  • Lane closures can only be confirmed on one specific topic.
  • Resuming an interrupted task needs the token from the interrupt.
  • Cancelling a task while it's interrupted leaves the robot stuck until a restart.

Each discovery became a rule in Auspex's adapter.

Telling a real stall from a normal wait. Early versions raised alarms for robots that were simply queueing for a working lift, and conflicts flickered open and closed every few seconds. I taught the monitor to recognise the waits Open-RMF explains, and to require a conflict to persist before acting on it.

Making the model dependable.

  • Nemotron Nano sometimes reasoned at length despite being asked not to, so its JSON answer was cut off and the interpretation failed closed. That meant the one conflict only interpretation can catch went unseen. Turning reasoning off in the chat template, and raising the answer limit, fixed it.
  • In another run, a delivery record arriving after a status update was dropped as "older", leaving a deadline from two days earlier. Nemotron then confidently chose handovers for deliveries that weren't late. Records are now merged, not discarded.

Both bugs showed the same lesson: the model is only as good as the picture it's given.

Accomplishments that I'm proud of

  • The complete loop working in simulation: detect, decide, gate, act through Open-RMF, verify, record.
  • A robot changing lifts mid-delivery because of information that only existed as free text in a hospital record.
  • A system that knows where its judgement should stop: clinical ties go to a person, causes are labelled "possible", and the benchmark results are labelled as results on an edge-case set.
  • A live site where anyone can replay the decisions, check the ledger themselves, and try to trick Nemotron, without being able to move a robot.
  • Grounding the problem in conversations with someone who deploys these systems, rather than my own assumptions.

What I learned

The hardest problem in hospital robotics isn't navigation, it's information. The facts that matter (who's waiting, how urgently, what's blocking the lift) live in systems that robots never see, and they're often incomplete and in free text. That's exactly where a reasoning model is genuinely useful, provided it's bounded by deterministic checks. I also learned to ask the people who deploy these systems before designing around assumptions.

What's next for Auspex

  • Levels of integration: observation only at any Open-RMF site; prediction where transport feeds exist; then hot-spot-aware planning from Auspex's own history.
  • Distribution through system integrators, who build this layer by hand for each site today.
  • Visual cause confirmation with NVIDIA Cosmos Reason, from robot or CCTV cameras.
  • Learning from the ledger: calibrating arrival-time estimates, and eventually fine-tuning Nemotron on recorded decisions, always with human approval for any change to policy.

Results

Results

Results on my edge-case scenario set: synthetic cases built so that the structured fields alone don’t give the answer. They are not general fleet performance. There is one recorded run per cell (12 runs in total), from the final rehearsal on Nemotron via Nebius Token Factory and on Tavily. Full table: live site → Benchmark.

Scenario A: Open-RMF alone B: Auspex, rules only C: Auspex + Nemotron S1 “CT 2” bariatric bed Patient-transport conflict detected no no yes Oxygen robot’s wait at the lift 133 s 124 s 0 s (switched lifts mid-task) S2 lift fault E-17 (self-clears) Oxygen on time yes, 282 s spare yes, 226 s spare yes, 274 s spare (waited, citing the bulletin) S2 lift fault E-42 (technician needed) Oxygen on time no (robot waited 578 s) no yes, 386 s spare (handover) S3 two Stat patients, one lift Oxygen on time no (robot waited 428 s) no yes, 175 s spare (nurse approved the order) Speed: the deterministic fast lane decided in 36–73 ms (budget: 2,000 ms). Nemotron’s decisions took 6–21 s, and only in the cases rules couldn’t settle. Cost: Nemotron used about 1,100 tokens for S1 and 3,300–4,300 tokens for S2 and S3. Safety: all 12 runs passed their scenario’s ground-truth checks.

Notes on the figures:

“No” for A in S1 is the baseline. Open-RMF can’t see the patient transport at all, so the robot meets the bed at the lift. B detects nothing in S1 because “CT 2” and the ventilator note are free text, which rules can’t read. In S2 with E-42, B can’t tell a 2-minute fault from a technician call-out, so the oxygen is late.

Built With

Share this project:

Updates

Submission history