Contextual smart notes

Inspiration

I wanted a tool that makes it effortless to turn what I find on the web into clean, useful notes — without sending my data to third-party servers. Highlighting a paragraph, getting a crisp summary, or rewriting a quote into something more polite should be instant and private. Browser extensions often copy text into generic clipboards or cloud services; I imagined an extension that brings on-device, privacy-first AI directly to the browser so research, drafting, and knowledge capture are faster and safer.

What it does

Contextual Smart Notes helps you capture and refine snippets from any webpage. Key features:

Capture highlighted text or media from the page (text, images, audio/video sources).

Summarize, rewrite, or expand highlighted content using on-device models.

Translate and proofread selections.

Run arbitrary prompts or start an interactive on-device session (LanguageModel).

Save refined notes to a local IndexedDB history for later recall.

Developer-friendly mock mode for offline testing and CI.

It’s designed to run locally when possible (using Chrome’s on-device APIs) and fall back to deterministic, testable mocks when not available.

How we built it

High-level architecture: Content script (content.js) captures user selections and media interactions and sends them to the popup.

Popup UI (popup.html, popup.js) is the user-facing interface; it orchestrates feature detection, model creation, streaming handling, and rendering.

Persistence layer (storage.js) uses IndexedDB to store notes locally.

Mock modules (e.g., prompt-mock.js, proofreader-mock.js) emulate on-device APIs for development and automated tests.

Manifest (MV3) wires content scripts and popup, with permissions for activeTab, scripting, and storage.

Implementation highlights: Feature detection first: we check for API objects like LanguageModel, Writer, Rewriter, and prefer them when present.

Availability caching: we cache model availability to reduce repeated checks and show progress UI when models download.

Streaming-friendly UI: long responses are shown progressively (for *Streaming methods) so users get immediate feedback.

Safe rendering: Markdown rendering is available; HTML output is sanitized (we removed blocks). For production we plan to add a proven sanitizer (e.g., DOMPurify).</p> <p>Developer ergonomics: a “mock mode” checkbox loads local mock modules via the popup so CI and offline testing are easy; small Node tests run via npm test.</p> <p>A tiny technical note: for streaming we&#39;re consuming async iterators like: $$ \text{for await (const chunk of stream) { ... }} $$ so the UI updates incrementally as chunks arrive.</p> <h2 id="challenges-we-ran-into">Challenges we ran into</h2> <p>Fragmented platform support: on-device APIs vary by browser/build. Fix: robust feature detection, graceful fallbacks, and mocks for development.</p> <p>Popup lifecycle and messaging: push messages from content scripts only reach an open popup. Fix: the popup queries the content script for the last selection on open (a pull), and content script also pushes messages when the popup is open.</p> <p>Sanitization vs usefulness: model outputs can be HTML or Markdown. Rendering HTML is powerful but risky. We removed <script> tags and render Markdown with a minimal renderer — but a production-ready sanitizer is required.</p> <p>Streaming UX: partial chunks can be awkward to show if they break Markdown structure. We progressively render but fall back to the text area when content is not safe or coherent.</p> <p>IndexedDB quirks and asynchronicity: handling DB upgrades and ensuring the popup UI updates reliably required careful transaction handling.</p> <p>Testing on-device behavior: because on-device APIs aren’t always present in CI, we invested in mock modules and Node tests. This added maintenance but made automated tests possible.</p> <h2 id="accomplishments-that-were-proud-of">Accomplishments that we&#39;re proud of</h2> <p>Built a privacy-first, on-device-first extension that lets users refine highlighted content without requiring remote servers.</p> <p>Delivered a full UI with summarizer, rewriter, writer, prompt sessions, translator, and proofreader integrations.</p> <p>Implemented streaming handling so users get progressive results rather than waiting for full responses.</p> <p>Created robust mocks and Node tests that let us run CI-style checks locally (npm test exercises translator/detector/summarizer mocks).</p> <p>Added developer-friendly options (mock mode, options page) and saved-notes persistence with IndexedDB.</p> <h2 id="what-we-learned">What we learned</h2> <p>Designing for varied platform capabilities means planning multiple fallbacks and making the UX tolerant of missing features.Streaming APIs change how you think about the frontend: instead of “wait then render”, you often have to render incrementally and think about partial states.Safety-first rendering matters — even internal tools should not blindly inject HTML into the popup.Small, well-documented mocks are incredibly valuable for developer velocity and reproducible tests.Popup lifecycle in browser extensions is subtle; the combination of push (content-&gt;popup) and pull (popup queries content on open) is the most robust pattern.A short example of something we documented during development: if $n$ is the number of selected sentences and we take at most $k$ sentences for a mock summary, the simple extraction behavior is effectively $O(n)$ to scan and select sentences.</p> <h2 id="whats-next-for-contextual-smart-notes">What&#39;s next for Contextual smart notes</h2> <p>Short-term priorities Add a trusted HTML sanitizer (e.g., integrate DOMPurify) and make rendered HTML safe-by-default.</p> <p>Expand mock coverage and add unit tests for writer/rewriter/proofreader mocks; run them as part of npm test.</p> <p>Add a background service worker to centralize the last selection so the popup can always fetch the last selection across tabs.</p> <p>Improve the proofreader UI to allow &quot;apply suggestion&quot; actions inline (accept/reject edits).</p>

Built With

• chrome.storage.local
• ci
• css3
• html5
• indexeddb
• javascript
• manifestv3
• node.js
• npm
• testing