Inspiration
Every operations team has an expert who handles the hard cases the way nobody else does — intuitively, correctly, invisibly. When teams try to capture that expertise, they reach for screen recorders. And screen recorders are genuinely good at half the problem: they capture the path. What they cannot capture is the moment the expert glanced at the VIP badge, glanced at the incident history, and decided this case was an exception.
Everything downstream inherits that gap. A macro replays clicks against a screen that has since changed. An RPA script encodes a path, not a reason. An LLM agent handed the same recording will happily invent a rationale, sound confident, and be wrong in exactly the place it costs you money.
The question that motivated this project: what does an agent do with the part of a recording that does not determine the answer?
What It Does
Show Me the Clicks watches one screen recording of a support expert handling a messy escalation case and produces a versioned, replay-tested, verifiable playbook — then runs the next case against it.
The flagship case: a VIP customer with a third delivery failure. The expert escalates, applies a tag, and issues a $25 goodwill credit. The recording proves VIP status was visible and two prior delays were visible. It does not prove which one caused the escalation — or whether both were required. Three different rules fit the same recording perfectly:
account_tier == VIP→ escalateprior_delays >= 2→ escalateaccount_tier == VIPANDprior_delays >= 2→ escalate
They agree on every frame and disagree on the next case that walks in the door.
So Show Me the Clicks refuses to pick. It surfaces the ambiguity as a first-class graph state, asks the expert exactly one question (the one with the largest downstream blast radius), and stores the rationale with the rule.
Then four things happen that matter more than the model:
Immutable versions — the corrected rule creates a new draft with its own content hash. Nothing is edited in place. Every past decision is reproducible against the exact rule that produced it.
Replay before publish — the draft cannot be promoted until it survives replay against the taught case and counterexamples. The naïve OR rule fails the near-miss (#2091: VIP, one prior delay). The AND rule routes it back to standard. Three green predicates → Publish unlocks.
Hard approval boundary on money — the $25 credit is a human-input node in the typed graph, not a prompt saying "please ask first." There is no edge in the graph from the model to that action without a human approval token.
Read-back verification — the runtime reads CRM state back and checks four independent predicates after execution. Model text is a claim. A read-back predicate is evidence.
The demo also deliberately breaks the assignment mid-run (failure-injection toggle) to prove idempotency: every mutating tool carries an idempotency key derived from execution ID, node ID, tool name, and version. The retry reuses the same key; exactly one receipt survives.
Best moment: the near-miss counterexample. Same VIP customer, one prior delay instead of two. The system does nothing — no escalation, no credit — and a verified predicate reads "no credit proposed." The impressive behaviour is the restraint.
How We Built It
The organising principle: keep probabilistic interpretation and deterministic execution strictly apart.
| Layer | Technology | Role |
|---|---|---|
| Multimodal extraction | Gemini 3.5 Flash | Aligns video frames, narration, and business data into timestamped observed steps with evidence ranges |
| Workflow graph | Google ADK 2 | Compiles observations into a typed graph with branches, tool nodes, and human-input nodes |
| Backend API | FastAPI on Cloud Run | Hosts all endpoints; stateless and horizontally scalable |
| State & versions | Firestore | Immutable versioned rules, execution receipts, audited state transitions |
| Video storage | Cloud Storage | Holds uploaded recording bytes |
| Billing & entitlements | Polar | Subscription checkout, webhook-driven pro status; independent admin allowlist as second path |
| Frontend | Next.js / React / TypeScript / Tailwind / shadcn | Full typed UI covering upload, extraction, evidence ledger, graph view, clarify, replay, publish, and runtime operator |
| Demo pipeline | Playwright + Remotion + Edge TTS | Drove the real running app scene-by-scene, captured bounding boxes, animated a spring-physics cursor, narrated with neural TTS |
Extraction pipeline (Gemini): Gemini receives video frames alongside screen-OCR data and narration transcript. It emits timestamped observed steps — each step carries a confidence score and an evidence range (the video interval that grounds it). The evidence is stored separately from the inference, so "this was visible at 00:14–00:19" is treated as fact while "therefore the expert believed X" must survive clarification and replay.
Safety invariant on observed text: every string lifted from the video (ticket bodies, customer names, field values) is treated as a data value, never as a directive. Prompt injection via attacker-controlled ticket content is structurally prevented.
Graph compilation (ADK 2): Observations compile into a typed workflow graph — seven nodes, one conditional branch, one human-input node. The split is explicit: reads and low-blast-radius writes run autonomously; money movement requires a human approval token. The model never gets to be the control plane.
Replay evaluator: Three test cases are evaluated against the draft rule before promotion is allowed. The OR/AND distinction that is the hardest learned-workflow bug is caught in under a second by a three-case suite.
Runtime operator: Picks up a published version against a fresh ticket. Reads, evaluates the branch, hits the injected failure, retries with the same idempotency key, applies the tag, saves the draft, proposes the credit — then stops and waits for approval. After approval: read-back verification of four predicates, four tool receipts, and the ADK execution trace.
Challenges We Ran Into
The ambiguity is architectural, not incidental. The hardest design decision was accepting that ambiguity is a first-class graph state and building the entire clarification flow around surfacing exactly the uncertainty with the largest downstream blast radius — not the first one the model noticed, and not a confidence score buried in a tooltip.
Immutable versioning with replay gates. Getting the version → draft → replay → publish flow right required careful Firestore schema design. Published versions are content-addressed; a clarification creates a child version, never mutates the parent. The replay evaluator had to run against the exact rule in the draft, not the currently-published one.
Idempotency in practice. Demonstrating that retries don't duplicate work required building an idempotency key derivation scheme (execution ID + node ID + tool name + version hash), storing receipts in Firestore keyed by that, and surfacing the "second attempt used the same key" trace visibly in the UI so judges could see it working — not just trust that it did.
Grounding vs. inference. The evidence ledger distinguishes what was observed (pixels at a specific timestamp) from what was inferred (what the expert believed). Keeping those two piles completely separate in the data model — and in the Gemini prompting — took several iterations.
Demo pipeline. Driving the real running app with Playwright, recording bounding boxes per scene, generating neural TTS narration, measuring audio durations with ffprobe, and building a Remotion composition that threads a spring-physics cursor through all 19 scenes with an audio-driven timeline was a significant engineering effort. Paywall dialogs intercepted clicks; nav locators needed RegExp matching; the first narration pass ran 233 seconds and had to be rewritten to 176 seconds.
Accomplishments That We're Proud Of
The near-miss counterexample is real. The system genuinely does nothing on the VIP-with-one-delay case. Not hardcoded — the typed graph evaluates the learned condition and routes to the standard path. The restraint is the feature.
Replay catches the bug that would have shipped. The OR rule passes the taught case and fails the near-miss. That failure is caught automatically, before the user can click Publish, by a suite that runs in under a second. This is the entire argument for treating learned workflows as versioned software artefacts.
Idempotency is visible, not asserted. The demo injects a real failure on the first assignment attempt, then retries. The action stream shows both attempts, the identical idempotency key, and the single surviving receipt. You can watch it work.
The approval boundary is structural. There is no path in the graph from the model to the credit action without a human approval token. This is not prompt engineering; it is graph topology.
The demo pipeline itself. A Playwright-driven storyboard of the real running app, a 19-scene Remotion composition with spring-physics cursor animation, neural TTS narration measured to the frame — all documented and reproducible in four commands.
What We Learned
Make ambiguity a state, not a score. A confidence number tells you the model is unsure. A human-input node tells you what to do about it. Only one is actionable.
Rank unknowns by downstream impact. You get roughly one question before an expert disengages. Spend it on the uncertainty that changes future behaviour, not the first one you notice.
Version the rule, not the prompt. Immutable versions with content hashes mean every past decision is reproducible against the exact rule that produced it. Prompts that drift are unfalsifiable.
Test learned rules like code. The OR/AND bug is not exotic — it is the single most likely way a learned workflow goes wrong. A three-case replay suite catches it in under a second.
Put the fence in the graph, not the prompt. "Please ask before spending money" is a suggestion. A human-input node with an approval token is a boundary.
Verify by reading state back. If your only evidence that the work happened is the model saying so, you do not have evidence.
What's Next
Real CRM connectors. The synthetic CRM is deterministic by design for the hackathon. Audited first-party connectors (Salesforce, Zendesk, ServiceNow) behind a read-back adapter would make this production-ready.
Multi-video learning. Currently one recording → one playbook. Multiple recordings of the same case type should let the system identify consistent patterns and surface conflicts between expert judgements.
Replay suite expansion. Today the evaluator runs three hand-authored cases. A generative replay pass — the model proposes adversarial counterexamples and the suite grows automatically — would catch more edge cases without expert authorship.
Explainability export. The evidence ledger and receipt store already have everything needed to generate a human-readable audit trail per execution. A one-click PDF export would make this useful for regulated industries.
Operator dashboard. Aggregate across executions to surface which branch conditions are most frequently hit, which cases most often reach the approval boundary, and where the human-in-the-loop overhead is highest.
Built With
- google-adk
- google-cloud-run
Log in or sign up for Devpost to join the conversation.