PocketDischarge

Your discharge papers, decoded on your phone in airplane mode. An Arm-optimized INT4 LLM gives you a plain-language summary, a jargon glossary, and questions to ask your doctor. Full Privacy.

Comment

Category (Mobile AI)

Inspiration

A discharge summary is the one medical document almost everyone receives and almost nobody can read. It is written by clinicians, for clinicians — "pt s/p PCI to LCx, cont. phenprocoumon, INR goal 2–3, VSD-occluder in situ" — and handed to a patient at the exact moment they are least equipped to parse it. People go home with a page of abbreviations and guess at their own care plan.

The obvious fix is to hand it to an LLM. The obvious problem is that a discharge summary is the most sensitive text a person owns, and pasting it into a cloud API means losing control of it permanently. "We don't store your data" is a promise; it isn't a guarantee.

That tension defined the project: the only version of this tool worth shipping is one where the note physically cannot leave the phone. And that turns a product decision into an optimization problem — can a model good enough to explain clinical language fit inside a phone's spare RAM and still generate at readable speed on an Arm CPU, with no accelerator and no network?

PocketDischarge is our answer, and we demo it in airplane mode.

What it does

You paste a discharge summary into an Android app. Entirely on-device, it returns three labeled sections:

  1. Plain-language summary — what happened during the hospitalization and what the plan is.
  2. Glossary — every abbreviation and piece of jargon in this specific note, expanded (LCx, CS, INR, phenprocoumon, VSD-occluder…).
  3. Questions to ask your doctor — five grounded, note-specific questions.

The engine is a LoRA fine-tuned Qwen2.5-1.5B-Instruct, quantized to INT4 (Q4_0), running under llama.cpp on the phone's CPU. On a Pixel 6 it occupies 935 MB on disk / 1.9 GB RAM, produces first token in 6.8 s and decodes at 9.1 tok/s, costing about 119 J per response measured off the battery rail.

The app's manifest declares no permissions at all — not even INTERNET. The privacy guarantee is structural and independently checkable, not a policy you have to trust.

How we built it

We ran it as a measured pipeline, with every claim tied to a number we could reproduce.

Data. We used Asclepius (arXiv:2309.00237), a corpus of synthetic-but-realistic clinical notes that is legally shareable — 158,114 rows across 8 task types. We kept the four tasks that map onto a patient-facing explainer (Summarization, Paraphrasing, Abbreviation Expansion, Question Answering) and dropped the four structural-extraction ones (NER, relation, temporal, coreference).

Verifying the schema produced our single most consequential finding: every row has a unique note, mapped to exactly one task. Notes are never shared across tasks. That meant splitting by patient_id gave a clean, leak-free note-level split (seed 42, 5% held out, frozen and reused across every later phase so numbers stay comparable) — but it also meant no gold three-part composite exists. We could only have built one by fabricating sections. We refused to, and the architecture below is a direct consequence.

Fine-tuning. LoRA (r=16, α=32, all attention + MLP projections, lr 2e-4, effective batch 16, assistant-only loss) on 32k examples for 2000 steps: 8040 s (~2.2 h) on an RTX 4070, peak VRAM 5.21 GB, loss 0.73 → 0.51. bf16 LoRA fit comfortably in 12 GB, so no QLoRA was needed.

The lift on the held-out 600-example split:

Model ROUGE-L BERTScore-F1
Base Qwen2.5-1.5B-Instruct 0.340 0.308
+ LoRA 0.490 0.522

Critically, the gains concentrate on exactly the patient-facing skills: Abbreviation Expansion 0.274→0.476, Paraphrasing 0.202→0.345 (ROUGE-L). The base model isn't stupid — it's verbose and stylistically wrong for the job. Fine-tuning fixed the register.

Quantization. We built the full ladder and scored quality at every rung on the same engine, so the comparison is honest:

GGUF Size (MB) ROUGE-L BERTScore-F1
f16 (reference) 3093.7 0.551 0.569
Q8_0 1646.6 0.548 0.566
Q5_K_M 1125.0 0.543 0.560
Q4_K_M 986.0 0.542 0.554
Q4_0 (Arm dotprod) 935.0 0.533 0.546

From f16 to Q4_0 the model shrinks $3.3\times$ for a quality cost of $0.018$ ROUGE-L and $0.02$ BERTScore. The fine-tune survives 4-bit almost intact.

The Arm-specific part. The Pixel 6's Tensor G1 is Armv8.2-A with NEON dotprod (SDOT/UDOT) but no i8mm, no bf16 matmul, no SVE, no SME2 — we confirmed this from /proc/cpuinfo before writing a line of optimization code, specifically so we would never claim a speedup from an instruction this chip lacks. The win therefore has to come from dotprod-optimized INT4 kernels, and we verified it live in the llama-bench -v load log: CPU_REPACK model buffer size = 329.90 MiB, repack ... with q4_0_4x4 / q8_0_4x4. llama.cpp repacks Q4_0 weights into the dotprod layout at load time. That repack is where Q4_0's speed advantage comes from.

Measurement. We cross-compiled llama.cpp with the NDK (arm64-v8a, GGML_NATIVE=OFF, -march=armv8.2-a+dotprod+fp16) and benchmarked with llama-bench (-p 512 -n 128 -r 3), sampling peak RSS from /proc/<pid>/status. Energy comes from integrating the battery rail directly:

$$E \;=\; \int V(t)\,I(t)\,dt \qquad\text{with}\qquad E_{\text{net}} \;=\; \int \big(P_{\text{load}} - P_{\text{idle}}\big)\,dt$$

sampling current_now and voltage_now at 4–10 Hz and integrating trapezoidally. Result: idle 0.49 W, load 7.15 W, 356.6 J over 768 tokens → 0.46 J/token net, ~119 J per response.

The app. A Kotlin Android app over a JNI bridge to llama.cpp, loading the GGUF from device storage.

Challenges we ran into

The full-precision model simply does not run. The Pixel 6 reports 7.4 GB MemTotal but only ~3.9 GB MemAvailable — Android is already using the rest. The 3 GB f16 weights plus compute and KV buffers exceed that, so llama-bench never loaded (RSS stuck at ~4 MB, 0 tok/s). This wasn't a setback so much as the project's thesis arriving as a hard error: on this device quantization is not an optimization, it is the difference between shipping and not shipping.

The model refused to produce three sections. A single composite "give me all three parts" prompt made the fine-tuned model collapse into just a summary. This is our own doing and entirely explicable: because no gold composite existed, we trained only on component tasks, so the LoRA specialized toward those behaviors and weakened adherence to a novel composite instruction. We had a fork — fabricate composite training targets, or change the architecture. We changed the architecture: the app makes three targeted calls, each with a prompt close to the training distribution, then assembles the output. It is more reliable, higher quality per section, and keeps every section grounded in something we actually trained and evaluated on real gold.

Measuring energy honestly was harder than measuring speed. dumpsys battery unplug lies — it changes the reported state but does not stop the hardware charger, so current_now never goes negative and you integrate the charger instead of the workload. The only correct method was to physically unplug the USB cable and keep adb alive over WiFi (adb tcpip 5555). That link then stalls on long runs, so we wrote integrate_energy.py --auto to segment load from idle by power level, making the result robust to a dropped shell.

A long tail of environment bugs, each of which ate hours:

  • import torch before from datasets import Dataset segfaults — a torch × pyarrow DLL ordering conflict.
  • Loading BERTScore's roberta-large in the same interpreter after a transformers CausalLM segfaults (reproducible on CPU too, so not CUDA). Fixed by splitting eval into two processes — which had the happy side effect of making predictions saved and re-scoreable.
  • str.splitlines() silently corrupted our JSONL, because real clinical text contains raw Unicode line separators (\x85, \u2028) that splitlines() treats as newlines and json.dumps(ensure_ascii=False) happily emits. Fixed with split("\n").
  • In-training sample generation drifted into Chinese gibberish — the model was left in train mode with dropout and gradient checkpointing active. In eval mode the same adapter produced clean, on-task English.
  • adb under Git Bash rewrote /data/local/tmp/... into C:/Program Files/Git/data/.... Every device script now sets MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*' and passes host paths through cygpath -w.
  • On-device llama-cli hangs (never exits after generation, block-buffered stdout loses output on kill). llama-bench is the correct non-interactive tool.
  • And twice, the mundane killer: the dev disk filled to 0 bytes mid-GGUF-write, and the phone's storage was 99% full so pushes failed silently. bench_on_device.sh now benchmarks one model at a time, deleting the previous GGUF before pushing the next, and records PUSH_FAILED instead of hanging.

Accomplishments that we're proud of

Deployed in ARM chip Every figure in this submission is measured on a real Pixel 6 or a real held-out split, and where we couldn't measure something we say so explicitly and show the analytic derivation instead. The f16 row in our results table is empty on-device, because f16 doesn't run — we left the hole visible rather than quietly substituting a PC number.

We published the results that made us look worse. Q4_K_M being slower than a larger model, and KV-cache quantization hurting generation speed, are both inconvenient for a tidy narrative. They're the two most useful findings we have.

The privacy claim is structural. Not a policy, not a toggle — the pipeline has no network path. We demo in airplane mode.

It's genuinely clone-and-run. Data prep, training, merge, quantization, cross-compile, on-device benchmark, and energy measurement are all scripted end to end.

What we learned

Smaller is not faster. Q4_K_M (986 MB) decodes at 7.49 tok/s while Q8_0 (1647 MB) manages 8.13 — a model 40% smaller is 8% slower. K-quant super-block dequantization costs more on this CPU than the plain dotprod kernels save. The real lesson: on Arm CPU, Q4_0 and Q4_K_M split the roles — speed vs. a sliver more quality — rather than one dominating.

KV-cache quantization is a memory lever, not a speed lever. Enabling it dropped decode from 9.08 to 7.80 tok/s and prefill from 75 to 51, because flash-attention plus per-token KV (de)quant has no hardware acceleration on Armv8.2 without i8mm or SME. You pay full price in software for a benefit you may not need.

Know your attention architecture before optimizing. Qwen2.5-1.5B uses GQA with only 2 KV heads across 28 layers at head_dim 128, so per-token KV is

$$2 \times 28 \times 2 \times 128 \times 2\ \text{bytes} \;=\; 28\ \text{KB/token (fp16)}$$

meaning even the full 32k context needs just 940 MB. Against 3.9 GB available, the weights are the binding constraint, not the KV cache — the exact opposite of the assumption we started with. We'd have wasted the whole phase optimizing KV if we hadn't done the arithmetic first.

Read the ISA before you write the optimization. Checking /proc/cpuinfo for asimddp, i8mm, sve, and sme on day one told us which optimizations were real on this silicon and which were marketing we'd have been repeating. Plenty of Arm-on-device writeups claim SME2 speedups on chips that have no SME2.

Constraints improve architecture. Being unable to fabricate composite training data forced the three-call orchestration, which turned out to be both more reliable and higher quality than the single-prompt design we originally wanted.

What's next for PocketDischarge

  • Speculative decoding with Qwen2.5-0.5B as the draft model — already validated on-device at 45.7 tok/s decode, so it's a plausible 1.5–2× on the 1.5B target with identical output.
  • ExecuTorch + XNNPACK + KleidiAI as a head-to-head against llama.cpp, to see how the Arm-blessed stack compares on the same device and quantization.
  • Newer Arm silicon. Every measurement here is from a chip without i8mm or SME2. On an Armv9 device with i8mm, the KV-quant result should invert from a penalty into a win — we want that number.
  • A mined abbreviation dictionary from the 19.7k gold Abbreviation-Expansion answers, applied as a deterministic grounding check on the glossary section so expansions can be verified rather than trusted.
  • Beyond discharge summaries — lab reports, imaging reports, and after-visit summaries are the same problem with the same privacy constraint.
  • Clinician review. Every quality number here is ROUGE-L and BERTScore against gold. Before this touches a real patient it needs clinicians rating real outputs for safety and faithfulness, and an explicit hallucination audit. Automatic metrics tell you the register is right; they cannot tell you the medicine is.

Built With

  • android
  • arm
  • arm64
  • c++
  • gguf
  • kleidiai
  • kotlin
  • llama
  • lora
  • neon
  • quantization
  • qwen
  • transformers
Share this project:

Updates

Submission history