Reality Diff — semantic memory for the physical world

---

The moment everyone has had

You are standing in a rental-car return lane, and the agent points at a scratch on the rear bumper. Was that already there? You took twenty photos at pickup, somewhere. You scroll. You cannot tell. You pay.

Or a friend asks when you replaced your desk chair, and you have no idea. The answer is sitting in your gallery, spread across fourteen months and eight hundred photos.

Photos are the richest record most people keep of their own lives, and that record is write-only. Photo libraries are excellent at finding pictures, but poor at reconstructing how the physical world changed over time.

What Reality Diff is

Reality Diff is a semantic memory for the physical world. It is a stateful Google ADK agent that turns a photo library into persistent, evidence-backed memory. It retrieves relevant photographs, asks clarifying questions when evidence is ambiguous, remembers user corrections across conversations, and changes future answers based on that feedback. Connect a photo library once, and it reconstructs the rooms, objects, vehicles, projects and plants that recur across your pictures as state over time, then answers questions about them in plain language with the source photographs attached.

In: your photos. Engine: three Gemini models behind a Google ADK agent. Out: an answer that opens back to a real photograph, a real timestamp and a real region.

Reality Diff doesn't wait for you to organize your gallery. During ingestion it filters irrelevant media, deduplicates photographs, extracts visible observations, embeds them, groups recurring subjects and reconstructs their state over time. When you ask a question, the ADK agent searches that already-built world model rather than treating your gallery as a bag of images.

Then you just ask.

"When did I replace my chair?" The dark mesh chair is last visible on June 4. The sand ergonomic chair first appears on June 11. There is no clear workspace photo between those dates, so the replacement narrows to that seven-day window. A single day is more than the photographs can prove. [View evidence →] June 4 photo · June 11 photo

You never built an album. You never tagged anything. Reality Diff discovered the "Home office" reality on its own, watched it change across fourteen months, and answered with the two photographs that prove it, naming the days it cannot see.

It tells you what it cannot prove

This is the feature that makes an answer worth keeping. Ask whether the rear-right scratch was there at pickup, and Reality Diff will not guess:

"I can't confirm whether the rear-right scratch was already there." The scratch is visible in a return photo. None of your pickup photographs shows the rear-right bumper. Absence from a region the camera never saw is not evidence that the mark is new. Then: "Add one pickup photo of that corner and I'll re-check."

Evidence-backed answers are constrained to retrieved photographs. Evidence IDs returned by the agent are validated against the retrieved set, and unsupported claims are downgraded to uncertainty rather than displayed as fact.

It builds your world for you

Connect a gallery and Reality Diff does the organizing. It discovers recurring realities and names them in plain language. Already available in the demo project:

• Home office: 42 photos across 8 months, with chair, monitor, lamp and plant changes bounded to the week
• White rental car: 18 photos assembled from a Barcelona travel burst, Aug 3 to Aug 8
• Blue bike restoration: 21 photos reconstructed as five stages
• Balcony pothos: 17 photos across 9 months of growth

The same world powers use cases you would otherwise build by hand: rental-car and rental-property evidence, project journals, plant timelines, leave-behind checks, disaster before-and-after. Reality Diff proposes them from your history, and you can start one deliberately. Both paths run the same evidence model.

A partner that leads, adapts and remembers

Reality Diff runs on the Google Agent Development Kit. The ADK agent owns the conversation: it decides which evidence tool to call, asks a question when the request is ambiguous, and carries state across turns.

• It asks before it assumes. Say "was this scratch already there?" and it finds two different marks, so it stops and asks which one, because choosing for you could produce the wrong claim.
• It follows the thread. Once it has asked which mark, "the rear one" is enough. The next turn resolves against the last, so a real back-and-forth works the way conversation does:

You: Was this scratch already there? Reality Diff: I found two marks on the white rental car, front-left and rear-right. Which one? You: The rear one. Reality Diff: I can't confirm the rear-right scratch was there at pickup. No pickup photo shows that corner.

• It learns how you think, and it changes the answer. Tell it "those two chairs are the same chair under different light," and the next time you ask about the chair it stops reporting a replacement and treats both photos as one chair. Corrections carry into every future question. No preferences wizard, no review queue.
• It separates memory from evidence. A correction can sharpen identity matching or ranking. It can never rewrite an original observation or invent a photo that was never taken.

Three Gemini models, each doing one job

Reality Diff never asks one model to do everything. A cheap model protects an expensive one, and deterministic code owns anything that must be exact.

Stage	Model / service	Job
Triage	gemini-3.5-flash-lite	Reject screenshots, memes and duplicates before they cost anything
Understand & reason	gemini-3.7-flash	Visible-facts-only observations and evidence-bound temporal answers
Retrieve	gemini-embedding-2	A shared 768-dimension image and question space. Vectors never leave the server
Orchestrate	Google ADK	The tool-bounded Collaborative Partner that runs the live conversation
Run	Cloud Run · Firestore · Cloud Storage · Pub/Sub	Scale-to-zero Google Cloud runtime

The invariant that makes it trustworthy: Gemini may interpret evidence, and it may never manufacture it. Every evidence ID the agent returns is validated against the retrieved set before a single pixel reaches the screen. Timestamps, hashing, duplicate detection and coverage are computed in code, never left to the language model.

It is live on Google Cloud

Open the app and ask. The service is public and serving:

$ curl https://reality-diff-284853036406.us-central1.run.app/health
{
  "status": "ok",
  "mode": "production",
  "model": "gemini-3.7-flash",
  "triage_model": "gemini-3.5-flash-lite",
  "embedding_model": "gemini-embedding-2",
  "live_models": true,
  "google_cloud_project": "reality-diff",
  "google_cloud_location": "global"
}

• Cloud Run service reality-diff in us-central1, scale-to-zero, health-probed.
• Vertex AI running all three Gemini models at the global location.
• The live /ask endpoint routes every question through the ADK agent, which selects the evidence tools and carries the multi-turn session.
• Firestore (delete-protected) for world state, Cloud Storage (public-access prevention) for originals, Pub/Sub with a dead-letter topic for ingestion.
• Least-privilege runtime identity, immutable image tags, and a project-scoped budget guardrail. The whole thing runs for a few dollars of model usage.

Upload your own photos in the live app and they are stored privately and analyzed by the real Gemini pipeline. Ask the demo's questions and you get the same evidence-linked answers the video shows.

Architecture

Web and native Android clients share one Google Cloud backend. Media flows through a staged ingestion pipeline (discover, filter, triage, understand, retrieve), builds a semantic world in Firestore, and is served to a Google ADK Collaborative Partner whose every answer links back to source photographs.

Features at a glance

• Zero-effort ingestion: connect an Android gallery (incremental MediaStore sync) or a browser folder (File System Access and drag-and-drop). New photos only, duplicates hashed out.
• Automatic subject and use-case discovery: recurring realities named in plain language, use-case collections assembled from bursts.
• Evidence-bound temporal reasoning: last-seen and first-seen intervals, explicit coverage gaps, and a "Why this answer?" trace that opens the source photos and the retrieval steps.
• Stateful, multi-turn dialogue on Google ADK: clarifications, follow-ups and corrections carry across the conversation.
• Persistent collaborative memory: aliases, identity merges and significance preferences learned through conversation, then applied to later answers.
• Native Android app: Kotlin and Jetpack Compose, Photo Picker, background sync, no WebView.
• Evidence by construction: missing coverage produces uncertainty, ambiguity produces a question, and every claim opens back to its evidence.

Reality Diff gives your photographs a memory. Connect them once, and ask your world what changed.

Built With

• fastapi
• firestore
• google-cloud
• jetpack
• kotlin
• python
• vertex