Swivel

Inspiration

A useful automation should survive the second run, the expired session, and the moment a human needs to step in.

Legacy software still makes people swivel between screens to complete routine work. Many of these systems have no convenient API. A computer-use agent can navigate them, but asking a model to rediscover the same job on every execution adds cost, latency, and uncertainty.

Swivel learns a job once, saves what worked, and replays it with checks and a human escape hatch. Built by Shivam Gupta, it explores a practical path from AI discovery to repeatable automation.

What it does

Give Swivel a goal such as reading a member's savings balance. During discovery, the AI observes a real Chromium browser, chooses actions through browser tools, and examines the resulting screen until the goal is complete. It saves the learned workflow as a typed, versioned capability with inputs, outputs, steps, assertions, policies, and provenance.

Later runs execute that capability without model calls. Swivel checks results, recognizes supported business outcomes, and uses bounded recovery for known failures. When it needs help, an operator can take control of the same live browser session, complete the difficult step, and resume automation.

The demonstration uses MERIDIAN, a synthetic legacy banking application, plus a second fictional institution to demonstrate tenant-specific overlays. All member records and credentials are synthetic.

Why it is agentic

The discovery loop is goal → observe → decide → use a tool → observe again → finish or escalate. The AI chooses the next action from current UI evidence; it does not merely return a conversational answer. The saved capability provides reusable memory of a successful workflow.

Execution then combines that learned knowledge with explicit contracts, failure detection, bounded recovery, and human handoff. A missing member is treated as a business outcome, while a session timeout can trigger a supported reauthentication path.

Demo distinction: the public website demonstrates deterministic execution of learned capabilities and recovery. It does not run an LLM in the hosted demo. The repository separately includes live AI discovery tooling and committed evidence from a real OpenAI model driving Chromium.

How I built it

The core is TypeScript on Node.js, with Playwright controlling Chromium and Zod validating structured artifacts. Live discovery uses model tool calls; the recorded reference run uses the OpenAI Responses API, and an optional Anthropic provider is also implemented.

An Express operator console exposes run state and evidence. Chrome DevTools Protocol screencasting and WebSockets support human control of the active browser. A control lease pauses automation while an operator owns the session. Firebase Hosting and Cloud Run serve the public demo.

The architecture separates discovery, capability storage, deterministic execution, and operator intervention. Tenant overlays let a shared workflow accommodate supported differences between fictional institutions. Per-run structured evidence and a hash chain help reveal changes to recorded events; they are not independent third-party attestation.

Challenges and lessons

The hardest part is deciding what a failed screen means. A valid “member not found” result should not become an endless retry loop. An expired session may be recoverable, while an ambiguous action may require a person. Swivel models those distinctions instead of treating every exception alike.

Human takeover also needs more than a screenshot: the person and automation must share session state, and only one should control it at a time. Preserving the browser and enforcing a control lease makes the handoff concrete.

The central lesson is that autonomy needs boundaries. Assertions, allowed tools, explicit outcomes, and escalation make a learned workflow more useful than a sequence of optimistic clicks.

What works today

The interactive demo covers five scenarios: reading a balance, a missing member, recovery after session expiry, execution for another institution, and human takeover followed by resume.

The repository includes recorded live discovery evidence: seven model turns produced a six-step capability, followed by a successful replay with zero model calls. In a fresh source audit for this submission, TypeScript checking passed and 192 tests across 46 suites passed. These checks validate the implemented system; they are not a claim of production banking certification.

Why this could be useful

The commercial hypothesis is straightforward: organizations with repeated tasks in legacy web systems could pay for reliable execution and exception handling. Reusing verified capabilities can reduce repeated inference, while evidence and operator intervention can make failures easier to diagnose. There are no claimed customers, revenue, or measured customer savings yet.

Try it and run it

Open the interactive demo and choose Open interactive demo. No signup or API key is required. Try the balance workflow, then recovery and human takeover. Watch the captioned walkthrough.

The repository README includes local setup, Chromium installation, demo scenarios, and live discovery instructions. The verified evidence documents the recorded model run and replay.

Scope and next steps

This is an existing project submitted as-is; it was developed before this competition submission. The current implementation is a browser automation system demonstrated against synthetic applications. Hosted storage is ephemeral, and production SSO, durable hosted persistence, and cross-process coordination remain future work. It should not be used for real financial operations as presented.

The next product milestone would be a narrowly scoped pilot on a permitted, low-risk legacy workflow, measuring completion rate, exception rate, operator time, and inference cost before expanding autonomy.

Built With

• node.js
• openai
• playwright
• typescript