Rulemetric.com

Train and keep up with all your agentic harnesses best practices in the background, passively. As insights appear, the agent gets smarter over time.

Comment

RuleMetric — the control plane for AI coding agents

Tagline: An x-ray and an eval harness for Codex — see exactly what your coding agent does, and measure which instructions actually make it better.

Inspiration

I hated that these things are black boxes. We hand Codex (and Claude Code, Cursor, Copilot) more and more control — and pour instructions into them through AGENTS.md, CLAUDE.md, and cursor rules — but you can't see what the model actually receives or produces, what it cost, or whether any of those carefully-written instructions helped at all. You're flying blind and guessing. I wanted to open the box.

What it does

RuleMetric captures every Codex session end to end and makes it legible:

  • See everything. Full context of every session, captured turn by turn — what you asked, what the agent did, which tools it called, how many tokens it burned — with an outcome grade on each session.
  • Measure your instructions. An A/B eval harness runs two instruction sets head-to-head against your real sessions and scores them, so you can finally say "this rule made the agent measurably better, this one did nothing — drop it."
  • Track cost & limits. Live usage against Codex quota (and Claude Code's 5-hour / 7-day windows), so you see spend and rate limits per tool.
  • Works everywhere you already work. One capture layer spans 25+ providers — OpenAI/Codex, Anthropic, Bedrock, Azure — and installs in about five minutes with an npm package.

How we built it

  • A lightweight two-layer HTTPS proxy sits between the coding agent and the model provider and captures the full request/response context without changing your workflow.
  • Session-capture hooks + a local capture gateway wire the agent (e.g. Codex / Claude Code) into RuleMetric on setup; a guided CLI (npm i -g @rulemetric/cli → rulemetric setup) diffs every change and masks secrets before it writes anything.
  • A background worker (launchd on macOS / systemd on Linux) long-polls the API and runs the analysis, insights, and instruction-effectiveness jobs.
  • A dashboard surfaces Sessions, Projects (grouped by repo), Usage, and Insights.
  • The capture pipeline stores raw LLM I/O, then parses and compresses it into structured, queryable session data.
  • Stack: Node.js 20.11+, [add the rest of your stack — API framework, datastore, dashboard framework, hosting].

Challenges we ran into

  • Getting people to care and use it besides myself. Developers won't change their workflow for a maybe — which pushed me hard toward a zero-friction, ~5-minute npm install and a setup flow that shows a diff before touching anything.
  • The firehose of text. LLM coding sessions produce an enormous amount of output. Storing the raw logs, then parsing and compressing them into something analyzable — without losing the fidelity that makes the capture worth having — has been the core engineering challenge.
  • Invisible-but-safe interception. Capturing full context across many providers and tools while never breaking the agent (a proxy/gateway that isn't listening can stop the agent from starting) took careful, defensive plumbing.

Accomplishments that we're proud of

  • Being able to see exactly what these LLMs produce — the black box is open.
  • Shipped a complete product solo: proxy, API, dashboard, npm CLI, and background worker.
  • One capture layer that actually works across 25+ providers, including Codex.
  • An eval harness that turns "which instructions help?" from a gut feeling into a score.

What we learned

  • The unmet need isn't more agent output — it's observability and measurement of it.
  • At LLM-session scale, storage and compression strategy is the product, not an afterthought.
  • Adoption is an install-friction problem as much as a value problem; every minute of setup costs you users.
  • Intercepting agent traffic cleanly across providers is full of edge cases you only find by running it on your own real work every day.

What's next for RuleMetric.com

  • Deeper Codex-native support — richer parsing of Codex/AGENTS.md workflows and quota.
  • An in-app instruction converter (canonical rules → AGENTS.md / .cursorrules / copilot-instructions.md), today CLI-only.
  • Team features — shared projects, org-wide instruction-effectiveness comparisons.
  • Sharper cost accuracy (per-model pricing) and the remaining live-rollup metrics.
  • Onboarding design partners — teams running Codex at scale who want to know which rules work.

Built with

node.js · npm · openai / codex · anthropic · http(s)-proxy · launchd · systemd · [+ your API framework, datastore, dashboard framework]

Try it out

Built With

Share this project:

Updates