DUET

DUET turns a spreadsheet and one successful demo into repeatable browser automation, helping office workers run web tasks without coding.

Comment

Inspiration

Many office workers, students, online sellers, and small teams still spend hours doing repetitive browser work from spreadsheets. They copy a value from a row, search it on a website, check the result, update the sheet, and repeat the same flow again and again.

This work looks simple, but it is mentally tiring and easy to get wrong. A small typo, a missed row, or an inconsistent action can create real operational problems.

DUET started from a simple idea: instead of asking an AI agent to freely decide what to do every time, what if a person could demonstrate the correct browser workflow once, and AI could help repeat that proven workflow across the rest of a spreadsheet?

What it does

DUET is a spreadsheet-driven browser automation assistant for repetitive web tasks.

A user prepares spreadsheet-like task data, performs the first task once in their real browser, and DUET records that successful demonstration as a repeatable workflow. DUET can then replay the same process for the remaining rows while logging actions, results, and failure points.

The core flow is:

  1. The user enters or imports rows of task data.
  2. DUET analyzes the sheet structure and identifies inputs, outputs, and repeated task patterns.
  3. The user completes the first row manually in the browser.
  4. DUET records clicks, typing, navigation, selections, and result-capture steps.
  5. DUET summarizes the workflow so the user can review it.
  6. DUET replays the workflow for additional rows.
  7. If something changes on the page, DUET uses stored element context and AI-assisted recovery to find the intended target again.

DUET is designed for people who understand their work but do not know how to write automation scripts.

How we built it

DUET is built as a full-stack browser automation system.

The frontend uses React, TypeScript, Vite, and Tailwind CSS for the dashboard, spreadsheet interface, workflow editor, and execution views. The backend uses Node.js, Express, Prisma, and PostgreSQL to manage workflow metadata, execution state, logs, and task records.

The browser automation layer uses a Chrome Extension with Manifest V3. It records the user's first successful demonstration inside the real browser and captures interaction details such as clicks, inputs, navigation events, and result logging.

For replay, DUET combines browser automation techniques with Chrome DevTools Protocol-style execution. Instead of relying only on fragile selectors, DUET stores richer context about each action, including element text, labels, roles, nearby content, structure, and input characteristics.

Gemini is used as an intelligence layer. It helps analyze spreadsheet intent, generate workflow summaries, explain recorded steps, and reason about recovery candidates when the page changes or a direct match fails.

Challenges we ran into

The biggest challenge was reliability. Real websites are messy. Buttons move, text changes, dynamic content appears, pages load at different speeds, and the same action can have slightly different results depending on timing or state.

A simple automation script may work once and then fail silently. For DUET, that was not acceptable. We needed replay to be inspectable, debuggable, and honest about failure.

Another challenge was balancing AI flexibility with user trust. General AI agents can be powerful, but they may reinterpret instructions differently each time. DUET takes a more controlled approach: the human demonstration is the source of truth, and AI assists with understanding, summarization, and recovery instead of freely controlling the entire task.

We also had to think carefully about privacy. Repetitive browser work often involves sensitive business data, customer records, or internal tools. DUET is designed around minimizing what needs to leave the user's local workflow.

Accomplishments that we're proud of

We are proud that DUET turns a familiar object, a spreadsheet, into the control center for browser automation.

Instead of asking non-developers to write scripts or design complex RPA flows, DUET lets them start with something they already understand: rows, columns, and one correct example.

We are also proud of the replay and evidence-focused design. DUET does not just try to automate a task; it records what happened, shows the workflow, logs execution results, and makes failures easier to inspect.

The project demonstrates a practical middle ground between traditional RPA and fully autonomous AI agents. It keeps the human in control while still using AI where it can provide real leverage.

What we learned

We learned that useful automation is not only about completing actions quickly. It is also about trust, repeatability, and clear evidence.

For workplace tasks, users need to know what the system did, where it got data from, what it changed, and why it failed when something goes wrong. A successful demo is not enough unless the system can explain and reproduce its behavior.

We also learned that AI is most valuable when it is placed in the right part of the workflow. In DUET, AI is not treated as an unpredictable browser pilot. Instead, it helps interpret spreadsheet structure, summarize workflows, and support recovery when deterministic replay is not enough.

Most importantly, we learned that the best automation tools for non-developers should feel less like programming and more like teaching by example.

What's next for DUET

Next, we want to make DUET easier to test with safe public demos and mock business workflows, so users and judges can evaluate the product without exposing private data.

We plan to improve the workflow editor, strengthen replay recovery, add clearer execution reports, and make it easier for users to approve or correct a workflow before running it across many rows.

We also want to expand the types of repetitive work DUET can support, including customer support operations, online seller workflows, research collection, platform review tasks, and internal admin processes.

The long-term goal is to make DUET a practical AI productivity tool for people who do important repetitive web work but do not have the time, budget, or coding background to build custom automation.

Built With

Share this project:

Updates