Inspiration
Lab reports are hard to read. Most people just want to know which numbers are out of range and what to ask their doctor. Pasting a photo of the report into a cloud chatbot is the easy option, but the same sheet also shows your full name, national ID number, medical record number, date of birth, address and phone number. We wanted the convenience of a large cloud model without sending that identity data anywhere. Hybrid inference lets us split the work: the device reads the document, and the cloud only sees anonymous numbers.
What it does
LabLens turns a photo of a lab report into a plain-language explanation, and keeps the identity data on the device.
- Upload or Try sample. Upload a JPEG/PNG, or load a synthetic sample (Indonesian hematology or US lipid panel; fictitious patients).
- Read & mask on the device. Local OCR (tesseract.js) plus deterministic PII detection masks the national ID (NIK), medical record number, dates, phone numbers, addresses, labelled names, and the report header/footer zones. If those zones cannot be found, cloud sending is locked (fail-closed). You see the redacted image and the masked text.
- Extract the results table.
- With Gemini Nano in Chrome, extraction runs fully on-device, and rows stream into the table as they are generated. No document data leaves the device.
- Without Gemini Nano, LabLens offers private cloud extraction, and only after you click to consent. It sends only the masked, result-shaped lines that pass a per-line allowlist; unknown test names become
Test Nand are mapped back locally. It never sends the image, header or footer.
- Flags computed locally. Low / high / normal comes from each row's reference range, computed in code rather than by the model.
- Explain in plain language (cloud). Gemini explains the anonymous rows in English or Bahasa Indonesia: a summary, a note per test, and 3–6 questions to ask your doctor. The medical disclaimer is fixed text in the code.
Two features make the routing visible. The Routing transparency panel shows each step's location (on-device or cloud), latency, bytes sent, and PII found on the device vs. PII sent to the cloud. The Payload Inspector shows the exact text that will be sent and what was sent.
How we built it
- Next.js 16 static export on Firebase Hosting, with React 19, TypeScript and Tailwind CSS 4.
- Firebase AI Logic (
firebase/ai) for all AI inference:getAI(app, { backend: new GoogleAIBackend() })- On-device extraction:
getGenerativeModel(ai, { mode: InferenceMode.ONLY_ON_DEVICE, onDeviceParams: { promptOptions: { responseConstraint } } })withgenerateContentStream. A JSON Schema constrains Gemini Nano's output, and a streaming JSON parser renders each row as soon as it is complete.initializeDeviceModel()warms the model up when the page loads. - Cloud fallback extraction and explanation:
InferenceMode.ONLY_IN_CLOUDwithgemini-3.5-flash-lite,responseMimeType: "application/json"and aresponseSchema(ExplanationResultfor explanations).
- Firebase App Check with
ReCaptchaEnterpriseProvider, enforced for Firebase AI Logic. - tesseract.js 7 OCR in a Web Worker with self-hosted WASM and language data (no CDN), under a strict Content Security Policy.
- A deterministic PII layer (
src/lib/pii/): regex and label rules, header/footer zone detection, canvas redaction, a per-field allowlist for cloud payloads, and anassertNoPiiguard that re-checks every cloud payload against generic patterns and against the PII values found in this document. - Vitest with 207 unit tests, and a playwright-core spike runner that drives real Chrome with Gemini Nano against synthetic fixtures with ground truth, to measure accuracy, latency, PII leaks and network traffic.
Challenges we ran into
- SDK session lifecycle. The hybrid SDK creates a new on-device session per call and has no way to abort an on-device request. We cache the model instance, warm it up on page load, and add our own timeout and cancel button. A cancelled Nano run still finishes in the background, which is a known SDK limitation.
- Gemini Nano latency. Output generation dominates. Total extraction takes ≈ 20–31 s in benchmark runs, and up to ≈ 36 s in the live app, depending on the machine. A leaner schema (flags computed in code, not by the model) and streaming rows brought the first row to ≈ 3.6–5.2 s in benchmark runs. In the live app, timed from the Extract click (so it includes Gemini Nano reading the image), the first row appears after ≈ 17 s, and the remaining rows stream in after that.
- PII in free OCR text. OCR output is messy: labels and values get split, and numbers look alike. Asking the model to find PII was not reliable enough, so masking is deterministic and fail-closed. Cloud payloads also pass a per-line allowlist and a final guard.
- App Check provider. reCAPTCHA v3 is deprecated for App Check, so we moved to reCAPTCHA Enterprise (score-based key) with debug tokens for local development.
Accomplishments that we're proud of
- On the synthetic fixtures, PII recall is 100% with 0 leaks (fixtures a, b, c).
- On-device extraction of fixture a produces 14/14 rows. The private cloud fallback produces 14/14 rows in ≈ 4.6 s.
- The Explain payload is ≈ 2.2 KB, with no image and no PII, and you can inspect it before it is sent.
- The privacy story is visible in the UI rather than just stated: the Routing panel and Payload Inspector show exactly what ran where and what left the device.
- The app works in any browser through the fallback, and you can try it without uploading anything personal (Try sample).
What we learned
- Gemini Nano works well for structured extraction from clean documents, but it should not be the only safeguard for privacy. Deterministic rules plus a guard are more predictable.
- Small on-device models do better with a smaller job: compute what you can in code (flags, placeholders) and keep the schema lean.
- Streaming changes how latency feels, even when the total time stays the same.
- Hybrid AI is not just "on-device or cloud". The useful design question is what data each side needs, and the minimum answer is often much less than the full document.
What's next
- Better accuracy on rotated or blurred phone photos. Fixture c is ≈ 61% today (deskewing, better table detection).
- A larger dictionary of test names and more report layouts and languages.
- Real cancellation of on-device inference once the SDK supports it.
- Trend view across several reports, stored only on the device.
Built With
- firebase-ai-logic
- firebase-app-check
- firebase-hosting
- gemini
- gemini-nano
- next.js
- react
- recaptcha-enterprise
- tailwindcss
- tesseract.js
- typescript
- vitest
