live lab
hero
learning
decision
verification
before after

Inspiration

Before writing software, I personally executed 3,000+ garment manufacturing orders. Today I build factory software for a working paint manufacturer in Punjab. Across both, the same failure repeats: critical procedures (changeovers, maintenance, quality checks) exist only as tacit knowledge in senior workers' heads. New workers learn by trial and error, and the cost shows up as rework, downtime and scrap.

Current SOPs don't fix this. They are slow to author, drift out of date, and cannot confirm that a step was actually performed correctly. USTAAD closes that gap with one-shot procedure learning: capture the expert once, then verify every worker after that, automatically.

What it does

USTAAD runs a closed capture → guide → verify → log loop, using any ordinary camera (phone, webcam or fixed station camera).

1. Capture: learning from one demonstration An expert performs the task once. USTAAD automatically segments the recording into steps and stores, for each one:

a reference keyframe: what "done correctly" looks like
a region of interest (ROI): where on the machine to look
feature descriptors: used later for matching

One recording becomes a reusable, machine-checkable procedure, with no manual SOP writing.

2. Guide & Verify: a perception–decision–action agent A new worker performs the same task. For every step, USTAAD gives a voice instruction, waits until the action is complete, compares the result against the expert's reference keyframe, and then decides what to do next based on what it sees: advance, correct, or escalate to a human.

3. Log Every session produces an auditable evidence trail: a timestamped photo, a verification score and a decision for each step, shown on a supervisor dashboard.

How we built it

OpenCV 5 vision pipeline (the core engine)

Temporal segmentation: frame differencing and motion-energy thresholding with hysteresis detect when movement stops and the scene settles, i.e. a step has finished.
Spatial registration: ArUco fiducial markers with pose estimation give a fixed coordinate frame, so expert and worker views line up even if the camera shifts.
View alignment: ORB keypoints, brute-force matching with Lowe's ratio test, and RANSAC homography warp the live frame onto the reference frame, correcting for angle and position.
Photometric normalisation: CLAHE on the LAB luminance channel reduces false failures caused by shop-floor lighting changes.
Step verification: SSIM and histogram comparison inside each step's ROI score how closely the worker's result matches the expert's.
Decision: calibrated dual thresholds turn that score into pass, fail or uncertain, with an explicit "not sure" zone.

Frames are adaptively sampled: they are processed only around motion events, not every frame, which keeps latency and compute cost low.

Agentic vision loop The agent orchestrates five tools: verify_step, next_step, flag_error, request_human_approval and log_evidence. The OpenCV verdict directly determines the next tool call:

Pass → log_evidence → next_step
Fail → flag_error with the specific mismatched region → re-verify after correction
Uncertain → request_human_approval, a human-in-the-loop safeguard so the system never guesses silently

Every decision is written to a structured trace (frame ID, scores, tool called, latency), so any outcome can be inspected and replayed.

Cloud architecture on AWS

[Amazon EC2 / ECS]: containerised FastAPI backend running the vision pipeline and agent, with a WebSocket channel for live frames
Amazon S3: reference keyframes and per-step evidence images
[Amazon DynamoDB / RDS]: procedures, sessions and decision traces
Amazon CloudWatch: latency, error rates and verification outcomes, for observability
[Amazon Polly]: text-to-speech voice guidance
IAM least-privilege roles: scoped access between services

Challenges we ran into

Viewpoint and lighting variance between expert and worker sessions, solved with fiducial registration, homography alignment and CLAHE normalisation
Threshold calibration: balancing error recall against false alarms so the system is strict without frustrating workers
Step segmentation: telling a finished action apart from a pause mid-action
Failure modes we still observe: [e.g. strong specular reflections, hands occluding the ROI, large camera displacement after capture]. These land in the uncertain band and are routed to a human rather than passed incorrectly.
Shipping the full pipeline (vision, agent, cloud deployment and evaluation) within a tight build window

Accomplishments that we're proud of

One-shot learning: a single expert demonstration produces a procedure that can verify every future worker.
Vision-grounded verification: the system checks what physically happened, not a checklist the worker ticks.
Calibrated autonomy: it acts on its own when confident and escalates when it isn't.
Measured results. We built a ground-truth test set before tuning anything: one real [N]-step procedure and [N] trials, with each step labelled correct or deliberately incorrect.
- Error recall (wrong steps caught): [88]%
- False-alarm rate (correct steps rejected): [8]%
- Precision: [90]%
- Median end-to-end latency per verified step: [320] ms
- Expert recording → usable procedure: [6] min

What we learned

Classical computer vision (feature matching, geometric registration, structural similarity) is fast, explainable and inexpensive enough to verify physical work in real time. Wrapping it in an agent that acts on the result turns a static check into a working assistant.
Building the evaluation set first made every later design decision measurable instead of guesswork.
An explicit "uncertain" state matters as much as accuracy. A system that knows when to ask a human is one a factory can trust.
Responsible use is a design requirement, not an add-on. USTAAD verifies the task, not the person: it is built to support workers, not surveil them. Everyone filmed gave consent, no client or proprietary data is used, evidence is stored with access controls, and uncertain decisions always go to a human.

What's next for USTAAD

Factory pilot: real procedures at a working paint-manufacturing plant, benchmarked against the current verbal/paper method on error rate and task time
Multilingual voice guidance: Hindi and Punjabi for shop-floor workers
Wearable-camera support: fully hands-free, first-person operation
Learned anomaly detection: layered on top of the classical pipeline to handle more complex, deformable or multi-object steps
Procedure library: procedures shared across lines and plants, so one expert's knowledge scales to every site