Inspiration
Every edit destroys information. A crop loses its edges, a JPEG re-save loses detail, a grayscale conversion loses color. So explaining one image from another is asymmetric: making the child from the parent is cheap, while rebuilding the parent from the child means inventing what's gone. Follow the cheap direction across a pile of copies and you get the family tree. We wanted a tool that does that and never pretends to know more than the pixels say.
What it does
Give Lineage a pile of image versions (originals, crops, captions, re-saves, restyles, AI edits) and it reconstructs the family tree: which file is the candidate original, who derived from whom, and which edit separates them. Every decision ships with its measurements and an empirically calibrated confidence. When the pixels can't tell, it says inconclusive instead of guessing.
- Identify (/identify): upload any meme from your phone. Lineage places it in one of 38 documented meme families and shows the documented original with its source, date and links, plus the measured edit between that original and your copy.
- Compare (/compare): drop in two files. The verdict is parent-child, siblings, unrelated or inconclusive, with the signals, the decisive cue, conflicts, unavailable signals, measured AI-edit indicators, limitations, and the SHA-256 of both inputs. An object added and an object removed look identical from pixels alone, so Lineage doesn't claim a direction there.
- Live tree (/tree): a big-screen forest that rebuilds as people make edits from a tablet or their phone, with a Fool Lineage scoreboard checked against the hidden truth.
- Audit (/audit): an analyst workspace for newsrooms, trust and safety, and legal discovery. Cases, batch upload, add from URL, reliability tiers, SHA-256 footprints, Content Credentials (C2PA) read and validated, and HTML/CSV reports with an integrity stamp.
- Method (/method): every cost term explained, plus the Numbers table with two numbers for every claim.
How we built it
Each image gets a 64-bit pHash (plus a binarized CLIP code when CLIP is on). Hamming distance picks the closest candidates, so the expensive directed comparison only runs on likely relatives.
For a directed pair a → b, ORB keypoints (SIFT fallback) and RANSAC align a onto b, then a per-channel color fit runs. The cost charges for everything b has that a cannot supply: coverage, color-fit residual, JPEG blockiness gained, resolution gained, chroma gained from gray, entropy gained, compact content differences, caption removed, an edit-operation description-length prior, and a tiny specificity tie-break. Every term only penalizes the direction in which b would have gained information.
Our own Chu-Liu/Edmonds arborescence (cross-checked against NetworkX in tests) picks one parent per image through a virtual root that separates families. Each edge keeps both directional costs, the runner-up and the margin, and the margin maps to empirical precision from calibration runs.
Pairwise verdicts are gated. A direction resting only on a symmetric cue isn't claimed, an aligned pair whose colors don't fit after correction is reported as a re-rendered copy with an AI-edit indicator, and margins under 0.05 count as indistinguishable.
Stack: Python, FastAPI, OpenCV, OpenCLIP, SQLite and c2pa-python on the backend; React, TypeScript, Vite and D3 on the frontend; public access through Tailscale Funnel.
Challenges we ran into
Real web memes were much harder than our synthetic edit chains. Our first meme benchmark claimed a wrong parent 14% of the time and called many captioned copies unrelated. We fixed registration on captioned images, added a caption-side cue (right 30 of 30 times on the benchmark), and added gates for re-rendered copies. On generative AI edits, wrong-parent claims fell from 26% to 8%, mostly by abstaining (68 to 76% of the time on those edits, by design). We also found the meme ground truth itself is noisy: in about half the pairs, the "documented original" we could download is smaller than its variant. So we publish both the all-pairs number and the clean-truth number.
Hand tuning overfit early, so every cost change now has to hold up on held-out runs, and we logged the rejected experiments. One weight change raised accuracy but flipped a real caption pair backwards, so we reverted it. A wider search identified more memes but put 2 in the wrong family, so we reverted that too: a wrong family is worse than an abstention.
Accomplishments that we're proud of
- Synthetic edit chains, pHash-only: 75.0% of parent links on the tuned run, 67.8% and 73.3% on two held-out runs; all 6 roots found every run; the true parent in the top-8 candidates 98.9% of the time.
- Calibrated confidence with two columns: at margin 0.30 or more, 92% correct on synthetic chains (35/38), 56% on real web memes across all pairs (95/169), and 73% on pairs with clean truth (71/97). The lower number is the honest expectation for unknown files.
- Identify on 378 held-out meme copies: 78.8% correct family, 21% abstained, 0 wrong families, median 3.1 s.
- 133 backend tests, 9 frontend tests, and a smoke test against the live server, all green.
What we learned
Information loss is a strong signal for roots and direction, but weaker for picking the immediate parent among several valid ancestors. Evidence is part of the product: forward and reverse costs, margins and calibration make a result reviewable instead of merely plausible. And a benchmark on messy real data told us exactly where to fix the engine, which a polished synthetic demo never would have.
What's next for Lineage
Screenshot copies (a photo embedded small inside a page screenshot) need region or multi-scale registration; Identify gets 2 of 13 today. After that: finishing the generative-edit corpus, internet originals through reverse-image search, and audio and video plug-ins under the same evidence and abstention rules.
AI disclosure
We built Lineage with AI coding agents under our direction. Claude Code wrote the backend, core algorithms, evidence and calibration work, API, tests and docs. OpenAI Codex built most of the frontend and the explanation plug-in. Google Gemini via Antigravity generated the edited images in our generative-edit benchmark and researched the Gemini image API. We designed the approach, the architecture, the evidence model and the evaluation protocol, reviewed the output, and made the engineering and tuning decisions.
Log in or sign up for Devpost to join the conversation.