Inspiration

Video calls used to be how we checked someone was real. Not any more: real-time face-swap software plugs into WhatsApp, Meet or Zoom as a virtual camera, and scammers use it to pose as a relative, a partner or a boss, then ask for money. We wanted a tool a non-expert can use during the call, that tells them what to do next, and that is honest when it can't tell.

What it does

Unmask runs in a browser tab beside your call. You share the call window, and:

OpenCV 5 measures the caller's face during a 10-second passive scan.
An agent (Amazon Nova Lite on Bedrock, via Converse tool use) picks the physical challenge that best resolves what the measurements are unsure about: turn your head fully, pass a hand across your face, or move towards a light. It shows the challenge as a prompt for you to read out.
Unmask measures the response, updates the scores and repeats (at most four challenges).
It gives a cautious verdict (Low risk, Suspicious, High risk or Inconclusive) in plain language, with clear advice: "Don't send money; call them back on a number you know."

Analysts can open the technical view: every signal, annotated evidence frames, and the agent's saw/chose/why trail, downloadable as JSON. Upload mode and one-click samples let judges try it without a call.

How we built it

OpenCV 5.0 does all the vision: YuNet face detection and landmarks, head pose via solvePnP, a face-boundary seam score, texture mismatch (Laplacian and FFT), hand-occlusion analysis (Farneback optical flow and MOG2), lighting consistency in Lab colour space, response latency, and quality gates.
A deterministic scorer turns these into the verdict. The language model never decides the verdict.
The agent makes one tool call per step (issue_challenge, request_better_conditions, conclude). Guardrails validate every call, and a rule-based policy takes over on errors.
AWS, all from one SAM template in eu-north-1: S3 + CloudFront, API Gateway, a Lambda container with OpenCV 5, Bedrock, DynamoDB traces with a 24-hour expiry, a private clip bucket that deletes after one day, and CloudWatch metrics with a dashboard. Images build in CodeBuild for a one-command deploy.

Challenges we ran into

Our honest result: the detector doesn't work yet. We recorded one participant in three lighting sessions and face-swapped each clip with an AI-generated face (an open-source swap tool at default settings). Evaluated leave-one-session-out, every strategy flagged 2 of 3 swaps and 2 of 3 real sessions, so there was no separation. The live demo agrees: the swapped sample ends Low risk.
Two pre-registered fixes, both failed. We wrote down new passive signals (resolution, sensor noise, skin tone, dropout) and the success criterion before computing anything; balanced accuracy was 0.46, against 0.50 before. A second idea, that the swap breaks during challenges, was wrong: the swapped face was more stable in all 6 pairs. We deployed neither.
Why: differences between lighting sessions were larger than the differences between real and swapped faces.
Engineering: messy browser video timestamps, OpenCV 5's slower DNN engine on CPU (fixed by detecting on a downscaled copy), Lambda cold starts (fixed with a keep-warm ping), and a very slow internet connection (we ran the swap job in a temporary private AWS stack and deleted everything afterwards).

Accomplishments that we're proud of

A real agent loop where OpenCV output changes the next tool call, with an exported trace that proves it.
Honest, pre-registered evaluation with controls, confidence intervals and failure cases, including a negative result.
Responsible defaults: 24-hour deletion, no face database, no identification, synthetic swap faces only, and a delete-now button.
Observability and reproducibility: CloudWatch dashboard, structured logs, and a one-command deploy.

What we learned

Modern face-swaps beat every classical signal we tested on this data.
The agent saved effort (2.67 challenges on average instead of 3), not accuracy.
Within the same recording, the swapped face was smoother in every pair, so comparing a caller against themselves looks more promising than fixed thresholds.
Limiting the language model to "what to test next", with a deterministic scorer for "what the evidence says", is a clean and safe design.

What's next

Many more participants, skin tones, devices, call apps and swap tools.
Paired, self-referenced checks instead of fixed thresholds.
Audio–lip-sync and voice-clone checks as extra challenges.
A browser extension that captures the call automatically, and an on-device passive scan for privacy.

Full technical report, results and code: https://github.com/breakthrough-child/unmask