AIMD Sentinel: Honest AI-Image Forensics

AI-generation result with the Advisory tag
manipulation mask, 6% of the image
strength of evidence and “Altered after signing” credentials
Nemotron investigation with its grounding check
forensic report

Inspiration

Every day, journalists, moderators and fact-checkers are asked one question: is this picture real? The tools they reach for usually return a single confident percentage, or a colourful heatmap nobody can interpret, with no account of how the answer was reached or how far to trust it. A detector score is not a probability, and a detector that has never seen a generator says nothing about its images. Overconfident tools do real harm: a wrong "authentic" stamp is worse than no answer.

We wanted a tool that behaves like a careful forensic examiner: measure, show where, and say how sure you are, and let an AI agent explain the evidence without ever being allowed to decide it.

What it does

AIMD Sentinel checks a photo (or video) and answers separate questions, each from detectors whose limits we measured:

Was it generated by an AI model? B-Free is the baseline detector that decides. UnivFD is shown for reference only and never votes.
Where was it edited? TruFor produces a per-pixel localisation map, which we cut into a manipulation mask drawn over the image (toggle Original / Masked).
How strong is the evidence? Each result is stated as a likelihood ratio fitted on a validation split, not a made-up confidence:

$$ \mathrm{LR} = \frac{P(\text{score} \mid \text{AI-generated})}{P(\text{score} \mid \text{real})} $$

When several kinds of check agree, the combined strength is reported as a range: one end treats them as fully dependent, the other as independent. That way we never claim more certainty than the data supports.

Is the file's history intact? C2PA Content Credentials are verified against the official trust list, so "altered after signing" is reported as a fact.
Investigation agent (NVIDIA Nemotron on Nebius Token Factory). nvidia/nemotron-3-super-120b-a12b reads the finished analysis through 8 read-only tools and writes a summary in which every sentence cites the tool call it rests on. Every number, URL and timestamp is checked against the tool results, and anything ungrounded is removed and listed. The agent can explain the result; it can never change it.
Reports. JSON, Markdown and HTML reports, with sha256 hashes of every artefact and a hash-chained audit log, so someone else can check the work.

Every verdict carries an "Advisory · not yet independently validated" label until the detectors pass independent evaluation.

How we built it

Backend: FastAPI (Python 3.11) running a pipeline: ingestion → preprocessing → detection (TruFor, B-Free, UnivFD, copy-move, C2PA) → fingerprinting → evidence fusion → reports. Artefacts are content-addressed, and queries are owner-scoped.
Agent: Nemotron 3 Super via the Nebius Token Factory OpenAI-compatible API with native tool calling. The tools have strict JSON schemas, there is a grounding checker, and limits are enforced from the database: 8 model calls, 60 s and 10 investigations per account per hour. The model never sees file paths, hashes or GPS data.
Calibration: likelihood-ratio calibrations are versioned and checksummed in the repo, and /health verifies that the running detectors match them.
Frontend: React 19, TypeScript and Vite. The result screen is deliberately simple: the AI-generation answer, the mask, the strength of evidence, Content Credentials, the investigation and the report.
Evaluation: labelled sets with a frozen holdout that no detector or rule was ever fitted or tuned on.

What we learned

Honesty is a feature. On the frozen holdout, TruFor caught 80% of controlled edits at an 8% false-alarm rate (AUC 0.91) and outlined the right area (mean pixel F1 0.87). But B-Free missed 45% of AI images from generators far from its training data. So we label results advisory, and "not indicated" never means "authentic".
LLMs are great explainers and dangerous judges. Giving Nemotron read-only tools plus a grounding check made its summaries useful and verifiable: hallucinated numbers are caught and stripped before anyone sees them.
A mask must agree with the verdict. Our first mask view sometimes highlighted areas the model itself had not flagged. We now gate the mask on the detector's own decision and say so in plain words.

Challenges

Calibration without cheating: fitting likelihood ratios on validation data only, and keeping a frozen holdout untouched for the final numbers.
Generalisation: AI-image detectors degrade on unseen generators, so we report that weakness instead of hiding it.
Grounding the agent: turning free-text LLM output into claims that can each be traced to a specific tool call, and flagging the ones that can't.
Running heavy forensic models on a 16 GB laptop: memory limits forced careful model loading and test isolation.

What's next

Independent evaluation of the detectors (the gate for dropping the advisory label), stronger detection of small inpaintings, and video manipulation support, which the app currently reports as "not usable yet" rather than "nothing found".