DriftSentinel

Inspiration

Enterprise teams spend thousands of hours maintaining UI automation workflows that break every time a web application is updated. A simple button rename or layout change can cascade into hours of debugging and manual fixes. We asked: what if automation could heal itself — just like how a developer would look at the old and new UI, figure out what changed, and update the scripts?

What it does

DriftSentinel is a self-healing UI automation platform that:

Runs existing automation workflows against a web app (v1)
Detects when the UI changes break the workflow (v2)
Analyzes exactly what changed — renamed fields, removed elements, new form fields, changed button flows
Repairs the workflow automatically by generating updated steps
Validates the fix by running the healed workflow in a sandbox

All with zero human intervention. In our demo, an expense form changed from v2.1 to v3.0 — fields were renamed, a Description field was removed, a Project Code was added, and a 1-click submit became a 2-step review process. DriftSentinel detected all 9 changes, repaired the workflow from 4 steps to 8 steps, and validated it with 100% success.

How we built it

We used three Amazon Nova models working together:

Nova Act — Browser automation agent that executes workflows against live web apps using Playwright, capturing screenshots and DOM state
Nova 2 Lite — Reasoning engine that analyzes DOM structural changes and generates repair plans with step-by-step instructions
Nova Multimodal Embeddings (Titan) — Computes visual similarity between v1 and v2 screenshots to quantify drift severity

The backend is built with FastAPI with Server-Sent Events (SSE) for real-time AI reasoning streaming. The frontend is a custom dashboard showing both app versions side-by-side with live pipeline controls. We parallelized page captures and embedding computations using ThreadPoolExecutor for faster detection.

Challenges we ran into

Nova Act handling complex instructions — Batching too many form fields into a single act() call caused timeouts. We found the sweet spot at 2-3 fields per call
Dropdown interactions — Nova Act struggled with HTML select dropdowns, requiring specific interaction patterns
SSE event matching — The frontend and backend used different SSE event names, causing streaming output to silently fail — took debugging to find the mismatch
Running on macOS vs WSL — Browser visibility (headless vs headed mode) behaved differently across platforms
Speed optimization — Initial drift detection took 12+ seconds due to sequential API calls. Parallelizing brought it down significantly

Accomplishments that we're proud of

End-to-end self-healing works — From detecting 9 structural UI changes to generating and validating an 8-step repaired workflow with 95% confidence
Real browser automation — Judges can watch Nova Act physically interact with web apps in real-time
Live AI reasoning stream — The dashboard shows Nova's thinking process as it analyzes changes and generates repairs
No product like this exists — Existing tools (Healenium, Mabl, Testim) only fix broken selectors. DriftSentinel handles structural changes like new fields, removed fields, and changed submission flows

What we learned

How to orchestrate multiple Nova models in a pipeline where each model's output feeds the next
The importance of prompt engineering for Nova Act — instruction specificity directly impacts reliability
Parallelization strategies for AI model calls using ThreadPoolExecutor
SSE streaming patterns for real-time AI output in web dashboards
That the gap between "smart locator fixing" and "true self-healing" is massive — and Nova's reasoning capabilities bridge it

What's next for DriftSentinel

Scheduled drift monitoring — Automatically scan target apps on a cron schedule and alert when drift is detected
Workflow versioning — Maintain a history of workflow repairs with rollback capability
Multi-page flow support — Handle workflows that span multiple pages and navigation steps
CI/CD integration — Plug into deployment pipelines to auto-heal workflows before they hit production
Visual diff overlay — Show side-by-side screenshot comparisons with highlighted change regions
Self-learning — Use past repair patterns to improve future repair accuracy and reduce healing time