AI Productivity Metrics

Zero-touch, local-first observability for agentic coding—connecting Codex missions, models, token checkpoints, Git activity, and outcomes into trustworthy, privacy-preserving metrics.

Comment

Inspiration

AI coding agents can work across repositories, spawn subagents, process very large contexts, run tests, and create commits. But after a task finishes, the evidence is usually fragmented across terminal logs, model metadata, token counters, Git history, and human memory.

A raw token total does not tell me which project benefited, what the agent actually delivered, whether the data is attributable to a specific mission, or how much uncertainty remains.

I started AI Productivity Metrics to answer a more useful question:

What did the agents actually deliver?

The project began as a small local metrics prototype. During OpenAI Build Week, Codex and GPT-5.6 helped transform it into a zero-touch, local-first observability system for agentic software engineering.

What it does

AI Productivity Metrics automatically connects evidence from:

  • Codex root threads and observable subagent relationships;
  • agentic missions, execution segments, interruptions, and resumes;
  • requested, configured, and observed models and effort levels;
  • reliable token checkpoints and comparable token deltas;
  • Git repositories, commits, files, and deliverables;
  • project discovery and stable repository identities;
  • synchronization freshness and zero-touch coverage;
  • subscription periods and longitudinal experiments;
  • backup integrity and tested disaster recovery.

The platform deliberately separates:

  • reliable evidence from diagnostic data;
  • period deltas from cumulative token inventories;
  • exact, probable, ambiguous, and unknown values;
  • observed facts from estimates and human interpretations.

Unknown data stays unknown instead of becoming a confident-looking guess.

The normal workflow is zero-touch: I use Codex normally, interrupt or resume sessions naturally, and occasionally launch a one-click synchronization. The system handles mission creation, project attribution, Git consolidation, checkpoints, reports, and verified recovery snapshots automatically.

How I built it

The core is written in Python and stores its structured state in SQLite.

Codex metadata is read from its local structured state in strict read-only mode. Git provides repository-grounded evidence such as commits, branches, files, and code changes. Hermes logs are preserved only as diagnostic sources when they cannot be reliably attributed.

Above these sources, I built a mission layer that links:

Codex root
→ project
→ mission
→ execution segment
→ checkpoint
→ reliable delta
→ Git deliverable

The active SQLite runtime remains outside synchronized folders under local application data. Reports and privacy-filtered recovery snapshots are published separately. Recovery snapshots are created through SQLite's official backup API, validated with integrity and foreign-key checks, hashed, filtered for sensitive fields, and periodically restore-tested.

The private production interface is generated locally. For Build Week, I also created a React and TypeScript frontend prototype with synthetic demo data. It presents the same core ideas through an Overview, Mission Explorer, Projects, Models & Effort, Evidence, Sync & Autonomy, and Backup & Recovery views.

The project now includes more than 500 automated tests covering migrations, concurrency, attribution, privacy, one-click synchronization, recovery, and zero-touch lifecycle behavior.

Challenges

Turning cumulative counters into honest measurements

Codex exposes useful cumulative counters, but a cumulative inventory is not automatically the consumption of a mission or time period. I introduced append-only checkpoints and only calculate a delta when two observations are genuinely comparable.

Attribution across projects and concurrent agents

Several agents can run at the same time, sometimes from different repositories. The system resolves only high-confidence relationships automatically. Ambiguous historical data remains visible but is excluded from reliable project metrics.

Handling interruptions and sleep

A terminal can stay open while the computer sleeps for hours. Wall-clock time, observable activity, and tracked segment duration therefore need to remain separate. Lazy reconciliation closes or resumes sessions without pretending that the entire elapsed time was active work.

Preserving privacy

The system is designed around local data. Public demos use synthetic data, and synchronized recovery snapshots pass a fail-closed privacy filter that removes raw paths, Codex identifiers, and free-text content.

Staying automatic without background noise

Early versions used periodic Windows tasks, which caused visible console popups. I replaced them with lazy, event-driven reconciliation and a silent one-click sync. There is no permanent server, no recurring watchdog, and no telemetry.

Accomplishments

I am especially proud that the project now:

  • tracks agentic missions rather than only isolated terminal sessions;
  • detects real Git projects even when they have no reliable token delta;
  • observes Codex models, effort levels, roots, and subagent relationships;
  • produces reliable token deltas without double-counting descendants;
  • distinguishes evidence quality and uncertainty throughout the interface;
  • survives network interruptions, sleep, concurrent agents, and resumed threads;
  • tracks subscription periods without merging incompatible baselines;
  • keeps its active database local while maintaining verified recovery snapshots;
  • restores and validates backups automatically without switching production;
  • remains usable with no regular manual tracking.

What I learned

The most important lesson was that measurement without provenance is often more misleading than having no measurement at all.

Tokens, elapsed time, lines of code, and commit counts can provide context, but none of them alone represents value. Reliable observability requires preserving where a value came from, what scope it covers, whether it is comparable, and what remains unknown.

I also learned that the right abstraction is not a terminal session but an agentic mission. A single mission may contain multiple segments, subagents, interruptions, models, commits, and deliverables.

Codex and GPT-5.6 were central to the development process. I used them for architecture audits, additive SQLite migrations, reliability reviews, Windows automation, privacy testing, frontend prototyping, documentation, and adversarial validation. My role was to define the product principles and make the key decisions:

  • zero-touch first;
  • evidence over anecdotes;
  • unknown over invented precision;
  • raw diagnostics separated from trusted metrics;
  • local-first privacy;
  • no permanent watchdog or cloud telemetry;
  • safe migrations and tested recovery.

What's next

The next step is to integrate the new React frontend with a sanitized Python-generated read model while preserving the current static fallback.

I also plan to:

  • collect larger experimental cohorts comparing model and effort configurations;
  • improve cross-platform support beyond Windows;
  • add portable encryption once a safe recovery-key strategy exists;
  • expand project- and mission-level comparisons without introducing opaque scores;
  • publish a polished public edition while keeping private runtime data fully local.

AI Productivity Metrics is not intended to decide which model is universally “best.”

It is designed to provide trustworthy evidence about which agentic workflows deliver value for a specific project, task, and engineering context.

Builder's note I built this project independently, and English is not my first language. The final video and public materials (also this text) were assembled under a compressed submission timeline, so they focus on the core engineering work rather than covering every feature in detail.

AI Productivity Metrics itself is functional, extensively tested, and still actively developed. I plan to continue improving the public presentation, documentation, frontend integration, and cross-platform support after Build Week.

Built With

Share this project:

Updates