Inspiration

Forms are deceptively hard. Real users (including friends abroad) kept asking “What does this field want?” on government, banking, and HR portals—often in a non‑native language and often asking for sensitive data. That leads to confusion, drop‑offs, and accidental PII leaks. With Chrome’s built‑in on‑device AI, we saw a chance to make a private, explain‑while‑you‑fill experience that works anywhere on the web.

What it does

Form Sherpa is a privacy‑first Chrome extension that:

Explains any focused form field in plain language with concrete examples. Translates labels and error messages to the user’s language. Autofills from a local, encrypted vault with masked previews and per‑site consent. Flags PII risk before submit and lets users redact or allow. Offers a “Form Map” side panel/popup to list all fields, roles, sensitivity, and one‑click “Go to” or “Autofill”. On-device by default, Cloud Assist as a fallback:

On-device: Uses Chrome’s Prompt API (LanguageModel) to generate structured outputs per field. Cloud Assist (opt‑in): If on‑device isn’t available (e.g., low disk space), it calls our tiny server that uses the Gemini API—never sending user‑typed values, only field metadata.

How we built it

Architecture (Manifest V3):

Content script (document context) Uses LanguageModel to analyze the focused field and return structured JSON. Renders an accessible microcard anchored to the field with “What this means”, examples, and a privacy note. Keyboard shortcuts (Alt+I), high‑contrast mode, and RTL awareness. Background service worker Settings and encrypted vault (AES‑GCM per entry; PBKDF2 key from a user passphrase). No Prompt API calls here (it’s not available in service workers). Options page Unlock vault, save values, set language, guardrails, and AI mode (Auto / On‑device / Cloud Assist / Lite). Side panel / popup (Form Map) Lists all fields on the page, shows detected role/sensitivity, and supports “Go to” and “Autofill”. Structured outputs (drives the UI predictably):

{
  "FIELD_SCHEMA": {
    "field_id": "fs_12345",
    "label": "National ID",
    "normalized_role": "government_id",
    "expected_format": {
      "pattern": "^[0-9]{9}$",
      "human_hint": "9 digits, numbers only. Example: 123456789"
    },
    "examples": [
      { "good": "123456789", "why": "Matches 9-digit numeric format" },
      { "bad": "123-45-6789", "why": "Dashes not accepted here" }
    ],
    "explanation_simple": "They’re asking for your national ID used for taxes/benefits.",
    "recommended_source": "vault.government_id_primary",
    "language": "en"
  },
  "RISK_REPORT": {
    "sensitivity": "high",
    "pii_types": ["government_id"],
    "suggested_actions": [
      "Verify the site is legitimate.",
      "Submit only over a secure connection."
    ],
    "redaction_rules": { "mask": "###-##-####", "alias_options": [] },
    "domain_context": { "domain": "example.gov", "is_https": true },
    "explain_risk": "Government IDs can enable identity theft if leaked."
  }
}

On‑device Prompt API details:

We use LanguageModel.availability() with the same expectedInputs/expectedOutputs we pass to create()/prompt(). When available, we request structured output via responseConstraint (JSON Schema) and omitResponseConstraintInput: true for efficiency. We cache results per (host + name + type + label + placeholder) to keep latency low. Cloud Assist (opt‑in, separate folder/server):

Node + Express + Gemini API (1.5‑flash), responseMimeType: application/json + responseSchema = our schema above. Rate limiting + LRU cache to reduce cost/latency. .env for GEMINI_API_KEY (never inside the extension). Input policy: send only field metadata (label/name/type/placeholder/nearby text + domain), never user‑typed values or vault data. Local validation example (credit‑card Luhn check used in guardrails): Let digits be d1,........,dn from right to left. Double every second digit, subtract 9 if > 9, sum all. Valid if S mod 10 = 0. $$ S= i=1 ∑ n s i ,s i ={ d i , 2d i −9 if 2d i

9, else 2d i ,

if i is odd from the right if i is even from the right $$

Challenges we ran into

Prompt API context constraints The API isn’t available in service workers, so we moved all prompting to the content script/side panel and messaged the background for vault/settings. Model download + hardware requirements Some machines lacked the ~22 GB free space needed for on‑device models; we added clear availability checks, a download progress toast, and the Cloud Assist fallback. Cross‑origin iframes and Permission Policy The Prompt API can be blocked in certain frames; we fall back to heuristics and still provide autofill/guardrails. Dynamic SPAs and shadow DOM Fields appear late or inside shadow roots; we added resilient field discovery and per‑field signature caching. Service worker lifecycle MV3 workers sleep; the in‑memory vault key disappears. We added an inline unlock prompt so Autofill can continue smoothly. ESM dependency quirks lru-cache v10 uses a named export (LRUCache), not default. Fixed import and documented setup. Windows DX differences Environment variable commands differ (export vs $env: vs set); we standardized on dotenv for a smooth dev experience.

Accomplishments that we're proud of

A cohesive, privacy‑first UX: explain, translate, autofill, and protect PII—all in a tiny microcard. Structured outputs that make the UI stable and trustworthy (FIELD_SCHEMA + RISK_REPORT). Full fallback ladder: On‑device → Cloud Assist (opt‑in) → Lite heuristics, so users on any device still get value. Accessibility from day one (keyboardable, ARIA roles, high‑contrast, reduced motion). A practical Form Map that scales from students to power users and teams.

What we learned

On‑device AI changes product decisions: with private, instant inference we can be proactive (show help on focus) without harming privacy. JSON schemas beat free‑text prompts for reliability: enums, constraints, and length limits reduce variance and simplify UI logic. Consent is a feature: clear “On‑device/Cloud/Lite” modes and chips indicating which path was used build trust. Engineering hygiene matters: caching, rate limiting, and token‑efficient prompts keep costs low and UX snappy. DX/UX documentation is as important as code: hardware/storage requirements and permission‑policy caveats need to be explicit.

What's next for Form Sherpa

Smarter guardrails: a full pre‑submit diff overlay with one‑click redaction, domain trust signals, and enterprise policies (force On‑device‑only, domain allow/deny). More validators: IBAN, GST/VAT formats across regions; smarter phone/address normalization. Learning‑friendly mode: show “why this example is good/bad” in more depth, and optional per‑site glossary. Cross‑origin coverage: handshake/helpers for embedded forms where Permission Policy blocks the Prompt API. Biometric unlock for the vault (where available) and richer value types (multi‑address profiles, alias emails). Packaging and distribution: Chrome Web Store listing, privacy policy, telemetry that’s strictly opt‑in and aggregate. Multilingual expansion: take advantage of new Prompt API languages as they roll out, plus inline TTS for accessibility.