Inspiration

Tax returns, bank account openings, and visa forms are full of jargon. When a field is confusing, the tempting shortcut is to paste the form — and the personal data already typed into it — into a general-purpose chatbot. That data leaves the machine, often into a prompt the user cannot see or retract.

The people this hurts are not looking for another chatbot. They want a plain-language note next to the question they are staring at, and they want the facts they already know (their legal name, their address, an ID number) to stay on their computer. Formwhisper started from that split: the explanation can travel to a model; the facts should not.

Hybrid inference is what makes that split practical. On a Chrome profile that already has Gemini Nano, the note can be written on the device. On a machine that does not, the same product can still help, but only after the person agrees to send the question — not the vault — to the cloud. The source of every answer is visible, so the product never pretends a cloud call was local.

What it does

Formwhisper is a Chrome side panel for difficult forms. Focus a field and it explains, in plain language, what the question means, why it is asked, and the mistakes people commonly make. The Page tab can build a checklist for a longer worksheet, or photograph the visible tab and read a scanned stamp as a JPEG — only after you ask.

Facts you already know live in a local vault (chrome.storage.local), optionally sealed with a passphrase (PBKDF2 and AES-GCM). The model never sees those values. It returns a semantic type from a fixed enum; Formwhisper matches that type to a vault key in code and, if you choose, places the fact in the box. A routing number or a passport expiry has no vault key, so nothing is filled. A privacy ledger counts on-device and cloud answers and keeps personal data sent at zero bytes, because vault values are not written into the prompt.

When Gemini Nano is available in Chrome, inference prefers the on-device model. When it is not, the panel asks for consent before any cloud call ("Send this question to the cloud?"). Never use cloud in Settings locks the product to on-device only. Every completed answer shows a Cloud or On device source chip. The demo video is recorded on a machine without Nano, so it shows the consent path and the Cloud chip honestly.

How we built it

All generative inference goes through Firebase AI Logic hybrid inference, following the web hybrid guide. The calls live in src/lib/hybrid-firebase.ts.

  • When Chrome already reports Gemini Nano as available, the field, checklist, and scan requests use PREFER_ON_DEVICE, so the SDK runs on device and can fall back only if the device path fails. We do not use PREFER_ON_DEVICE while the model is missing, because we do not want a silent fall-through to the cloud.
  • When Nano is not available, the panel shows a consent dialog and, if the user agrees, runs ONLY_IN_CLOUD. Cloud calls use the Gemini Developer API (GoogleAIBackend) on the Spark plan, with gemini-4-hydrogen as the cloud model. Structured JSON is required (responseMimeType: application/json and a response schema).
  • App Check is required by Firebase AI Logic and is enforced for these calls, with limited-use tokens. Manifest V3's script-src 'self' blocks the reCAPTCHA Enterprise script, so the unpacked side panel uses the App Check debug provider, which the hackathon rules accept. The debug token is registered in the Firebase console and stored only in gitignored config or in local extension storage. It is never committed.
  • On-device Nano is reached through the Chrome Prompt API (the hybrid SDK's on-device path), with a user-gesture model download from Model setup.
  • The UI is a Manifest V3 Chrome side panel (TypeScript, React, Vite). Opening the panel injects one self-contained content script into the active tab; pages with no fields get no listeners. The vault, matching, and fill path are ordinary in-extension code, not model tools.

Challenges we ran into

The first cloud answers failed with "The model did not return JSON." App Check was not the cause; a 128-token output limit on field calls was truncating the structured object. Raising the field limit to 1024 (and checklist/scan to 4096) made valid JSON come back.

On-device inference needed care too: Nano can take a while on integrated GPUs, so a fixed timeout killed valid runs. We switched to streaming with an inactivity timeout and a Cancel button, and we release on-device sessions when a tab closes so the GPU is not left busy.

App Check on a side panel was a genuine platform constraint. There is no Chrome Web Store attestation provider today, and reCAPTCHA Enterprise cannot load under the extension's CSP. The honest, rules-compliant path for an unpacked extension is the debug provider, with the token kept out of the repository.

Accomplishments that we're proud of

A complete product loop on a real form, not only a model hello-world: field notes, a page checklist, a visible-tab scan, a local vault, and a Fill button that never sends the fact to the model. Hybrid origin is visible on every answer. App Check is on the cloud path. The privacy ledger stays at zero bytes of personal data sent because the architecture does not put vault values in the prompt. The extension is tested for extraction, vault matching, JSON parsing, and hybrid wiring, and it is licensed Apache 2.0.

What we learned

Hybrid inference is not only a fallback flag. The useful work is in the product decisions around it: do not silently upgrade a missing on-device model into a cloud call; say so in the UI; keep personal facts out of the prompt even when the explanation runs in the cloud; and size structured-output limits so the model can actually return the schema you asked for. We also learned that App Check means something different inside a Manifest V3 side panel than on a normal https origin.

What's next for Formwhisper

Record the on-device path in the docs so both source chips are shown. Explore a Chrome Web Store release, knowing that App Check for a packed extension still needs a viable attestation story. As the product grows, every new feature keeps the same rule: the model may see the question; it should not see the vault.

Built With

Share this project:

Updates

Submission history