GIF

Inspiration

On a repeating semiconductor array (FinFET fins, DRAM capacitors, contact grids), a CD-SEM cannot tell which identical copy it is looking at. Measure the wrong copy and process-control data is silently corrupted, and nobody notices. Plain template matching sees dozens of equally good matches. We wanted a system that says how sure it is, and acts accordingly instead of guessing.

What it does

DriftSense Pilot localizes a reference patch on a periodic array and decides what to do next:

Frequency tells you where a copy could be; roughness tells you which one is real. A lattice FFT finds the candidate positions, and a line-edge-roughness fingerprint separates the real copy from its identical neighbours.
A calibrated uniqueness margin is compared with a threshold tau. At or above tau the agent commits the stage move and verifies. Below tau it re-acquires (double the dwell on low SNR), re-localizes in contour mode, and when its step and dose budget is spent it asks a human: an operator card shows the top three candidates, and the decision goes in an audit trail.
The same six tools (acquire, localize, refine, commit_stage_move, request_human_approval, get_trace) are exposed over MCP, so any MCP client can operate the virtual SEM. The "never commit below tau" gate is enforced in the runner and the MCP server.

Try it: pick a demo site and run an agent episode. The hard DRAM site escalates to an operator; the FinFET site commits on the first look.

How we built it

OpenCV 5 engine (stages S0-S9): ingest/QC, lattice FFT, rectification, roughness residual, candidate scoring with matchTemplate, cascade, subpixel refinement and calibrated confidence. Hot paths use KleidiCV-listed operations (GaussianBlur, Sobel, erode, dilate, warpPerspective, pyrDown, resize) so COOL can accelerate them on Graviton.
Virtual SEM digital twin used for every result, plus an active-perception agent (rules policy, safety runner, optional LLM planner) with every step traced to JSONL.
Web app: TypeScript single-page UI, FastAPI service and SQS worker.
AWS CDK (Python) infrastructure for ap-south-1: CloudFront, API Gateway and arm64 Lambda, SQS, DynamoDB, S3, a Graviton4 worker on the COOL 3.1 AMI, CloudWatch dashboards, alarms and a canary. The same app also runs free in one container (Docker, Hugging Face Space or Render).
Rigor: pre-registered sealed evaluation (protocol hash recorded before the run), docs rendered from results files, and a claims checker that fails CI if a number is not traceable.

Challenges we ran into

Periodic arrays are adversarial by construction: when two copies are genuinely indistinguishable in one frame, no estimator can pick the right one. Our answer is not a better guess but a better decision: abstain, look again, or ask a person. We also had to keep the evaluation honest. We found and fixed harness bugs on a development set, disclosed them, and kept the sealed set sealed.

Accomplishments that we're proud of

On the sealed, pre-registered episode set (48 episodes):

The agent succeeds on 100% of episodes, against 91.7% for single-shot localization, with 0% false commits.
The true site is in the candidate set on 96.7% of structured pairs.
OpenCV 5.0 agrees with OpenCV 4.12 on 100% of decisions.
Every number in the README and report is generated from versioned result files.

What we learned

Calibrated confidence matters more than raw accuracy when the cost of a silent error is high, and "ask a human" is a feature, not a failure. Re-imaging has a price: it uses more electron dose than a single shot (about 1.4x), and when two copies are truly indistinguishable the agent asks an operator. The full protocol and limitations are in the report.

What's next for DriftSense

A dose-aware policy that spends the extra imaging only where it pays, the DRAM contour branch (scaffolded, not yet on the live path), the COOL benchmark campaign on Graviton4 to measure the speed-up per stage, and real-SEM validation with an instrument vendor.