Automation Auditor

Inspiration

Does your automation tool really save you time? A quick automated response can still leave someone uploading files, checking the output, correcting mistakes and saving a usable result. Automation Auditor measures that human work instead of treating machine speed as productivity.

What it does

Automation Auditor compares the active human work needed to complete the same task manually and with an external automation tool. Users define the work, deliverable and completion conditions, control their own timing and confirm completion. Code calculates the difference from recorded events.

The four-step workflow is Automations, Benchmarks, Run Console and Compare. Users connect a tool and version, define a shared task, record manual and assisted work, then see the time comparison. A second automation version is optional. Users can import materials and answer three task questions, or supply a supported task package.

The actual task and its deliverable stay outside Automation Auditor. The app registers the saved result's location rather than uploading that output. Required operation, checking, corrections and exporting count as human work. Only compatible, completed records with user-confirmed full timing produce a normal comparison.

A downloadable Time Benchmark Certificate presents the tool, task, recorded times and completion references. It reports a user-confirmed time comparison, not independent output-quality certification. Audit Report adds supporting records and optional AI explanations without becoming a mandatory fifth step.

What we observed

One operator converted 12 synthetic PNG images into JPEGs, preserving content and aspect ratio, limiting the longest edge to 1,200 pixels and not enlarging small images. Manual work in macOS Preview took 12m35.930s. Work with Bulk Resize Photos took 9m29.850s, including the operator's processing, saving and checking. The recorded reduction was 3m06.080s, or 24.6% of active human work time.

Both records use the same inputs and completion conditions, and EddieM confirmed completion and full timing. This is one fixed-order observation with prior instruction, not a randomized study or a claim about the fastest manual method: the baseline processed images individually, while Preview also supports batching. The public replay kit lets reviewers measure their own result; they are not expected to reproduce 24.6%.

How we built it

The TypeScript and React application stores workspace records on the user's browser and device, fingerprints task files and preserves append-only events. Deterministic libraries recalculate time and reject incompatible comparisons. Result-reference history and evidence exports preserve the original submission and later valid changes. Cloudflare Workers and D1 host the protected model interface and budget ledger.

NVIDIA Nemotron, nvidia/nemotron-3-super-120b-a12b, runs through Nebius Token Factory. The Evidence Analyst explains recorded human work and proposes questions to investigate. It does not create timestamps, certify business quality or sign completion. Access protection, idempotency and an application budget bound model usage; deterministic measurement remains usable without AI analysis.

The video includes a real analysis of the same image-task records. The October 24 call completed with HTTP 200, a stop finish reason, 3,095 total tokens and 13,677ms latency. Its hypotheses are suggestions for further investigation, not established explanations of the time difference.

Challenges and lessons

User-led iteration made the measurement understandable to nontechnical users: a clear four-step overview, three task questions, a single place to open the external tool, explicit result registration and a shareable certificate. Separating measurement, completion responsibility and AI interpretation kept the product focused while making its evidence inspectable.

Existing work and attribution

Material Auditor is a separate pre-existing tool used in earlier document-review trials. Bulk Resize Photos is an external third-party website, not part of our product. Automation Auditor's connection and task workflow, event-based comparison, user-completion controls, local recovery, Nebius explanation path and certificate were developed or extended during this project. The repository records source scope, dependencies and notices; MIT attribution uses EddieM.

What is next

Repeat the image task with more operators, counterbalanced order and an efficient manual batching method. This would test how robust the observed time difference is without turning Automation Auditor into a business-quality scoring tool.