Screenshot-Search

Inspiration

We all do the same thing: screenshot something important, promise ourselves we'll remember it, and never find it again. Wifi passwords, booking confirmations, that book recommendation from a group chat three months ago, an OTP we screenshotted "just in case" — buried in a camera roll of 3,000 images with no organization.

The tools that exist today don't solve this well. Google Photos does keyword OCR, but it only matches exact text — search "wifi password" and it needs those literal words in the image. ChatGPT can read screenshots, but you'd have to upload them one by one, every single time, into a conversation that forgets everything. And both require sending your most sensitive data (bank statements, private conversations, medical reports) to someone else's server.

We wanted something that searches by meaning, works instantly across thousands of images, and never uploads a single byte. So we built it.

What it does

Recall is a local-first screenshot search engine that runs entirely in your browser. Drop in your screenshots — tens, hundreds, thousands — and Recall OCRs every image, understands the content semantically, and indexes it all on your device. Then you search in plain language.

The key difference from existing tools is semantic search. You don't need to remember the exact words on the screenshot. Type "that code my friend sent for the router" and Recall matches an image that says "Network: HomeWifi_5G, Password: xK9#mL2" — zero overlapping words, but it understands the meaning. Type "money someone owes me" and it surfaces a screenshot of a Splitwise balance or a WhatsApp message saying "I'll pay you back Tuesday."

Smart filters auto-detect content types — URLs, code snippets, phone numbers, prices — so you can narrow results without thinking. Everything persists in IndexedDB, so your index survives browser restarts. You index once, search forever.

Nothing ever leaves your machine. No accounts, no cloud, no API calls. Your screenshots stay yours.

How we built it

The entire app runs client-side with no backend:

React + TypeScript + Vite for the application shell Tesseract.js running in web workers for OCR — extracts text from every screenshot without blocking the UI @xenova/transformers running the MiniLM-L6-v2 model directly in the browser (via WASM) to generate semantic embeddings from the extracted text Dexie (IndexedDB wrapper) to persist images, OCR text, and embedding vectors locally — the index survives page reloads and browser restarts Hybrid search engine that blends Fuse.js fuzzy keyword matching (0.45 weight) with cosine-similarity semantic search (0.55 weight) for results that catch both exact and meaning-based matches Tailwind CSS + Framer Motion for the UI, with shared-element layoutId transitions, animated grid reflow, and a full dark/light theme system

The processing pipeline per image: generate thumbnail → OCR via Tesseract worker → clean extracted text → compute semantic embedding → store everything in IndexedDB. A concurrency-limited queue (2 workers max) handles bulk imports without choking the browser.

Challenges we ran into

Tesseract.js performance. Client-side OCR is not fast. Our first attempt processed images sequentially and a batch of 100 screenshots took over 10 minutes. We moved OCR into web workers with controlled concurrency, added thumbnail pre-generation so the UI stays responsive, and implemented a processing queue with live per-image status updates so users aren't staring at a frozen screen.

Running ML models in the browser. Loading a transformer model client-side has real constraints — initial model download is ~30MB, inference isn't GPU-accelerated on all devices, and memory usage spikes with large batches. We had to cache the model aggressively, batch embedding computations, and implement query embedding caching so search stays under 50ms after the first keystroke.

Hybrid search scoring. Blending keyword and semantic results sounds simple until you realize the score distributions are completely different (Fuse returns 0–1, cosine similarity returns -1 to 1 in theory but clusters around 0.2–0.8 in practice). We had to normalize both score ranges before blending, and the 0.45/0.55 weighting took real experimentation to get results that feel right.

IndexedDB storage limits. Browsers have storage quotas, and storing thousands of image blobs plus their thumbnails plus embedding vectors adds up fast. We had to handle quota-exceeded errors gracefully and compress thumbnails aggressively to keep the footprint reasonable.

Accomplishments that we're proud of

The semantic search genuinely works — you can describe a screenshot in your own words and find it, even when none of those words appear in the image. Demoing this to people reliably gets a "wait, how did it find that?" reaction. Zero data leaves the device. No API keys, no backend, no accounts. We proved you can build a useful AI-powered tool that respects privacy by default, not as an afterthought. The UI feels like a real product: shared-element lightbox transitions, animated grid reflow during search, a polished theme toggle, keyboard shortcuts throughout. We didn't want it to look like a hackathon project. Handles bulk imports of 100+ screenshots without freezing the browser — the processing queue with live status updates makes even the waiting feel responsive.

What we learned

Running ML models client-side is viable but requires serious thought about loading, caching, and memory — you can't just import a model and call .predict() like you would server-side. Hybrid search (keyword + semantic) is significantly better than either alone. Pure semantic search hallucinates relevance on short queries; pure keyword search misses everything that isn't an exact match. The blend catches both. Privacy-first architecture is a genuine design constraint, not just a marketing angle. Every feature decision starts with "can we do this without a server?" and sometimes the answer forces creative solutions. Framer Motion's layoutId for shared-element transitions is absurdly powerful — it took one prop to create the kind of animation that would take hundreds of lines of manual code.

What's next for Recall

Browser extension that auto-captures and indexes screenshots as you take them — no manual import step needed. Cross-device sync using encrypted peer-to-peer transfer (no central server), so your index is available on your phone and laptop. Smarter content extraction — detect and parse structured data like tables, addresses, and calendar events from screenshots, not just raw text. Natural language actions — "find the tracking number and copy it" or "find the event details and add to calendar" — moving from search to workflow. Desktop app via Tauri for native file system access, system-wide search shortcut, and better performance than the browser sandbox allows.

Built With

• react
• typescript
• vite