• 

  Clerecy AI Router Selecting A Golden Submission

Clerecy AI Router

The control plane that learns which AI model earns each job.

Inspiration

Clerecy builds applications for regulated work.

RIFT CMS helps content teams create and modify governed websites with an embedded AI agent. UnionNexus helps union representatives capture the who, what, when, where and why of workplace grievances.

Both applications require a named person to review AI-generated work before it becomes official. But behind the scenes, they had the same conventional architecture: one provider configured in an environment variable, with almost every request sent to a flagship model at flagship prices.

That left us unable to answer three questions:

• Which model is the cheapest one that is still reliable for each task?
• Which models perform best on our actual workloads—not public benchmarks?
• Why were we throwing away the human review decisions that already told us whether an answer was useful?

Clerecy AI Router turns those review decisions into a continuously improving model-control loop.

Other routers know what models cost. Clerecy learns which models earn each job.

What it does

Clerecy is an OpenAI-compatible gateway and autonomous control plane for applications and institutional AI agents.

An application changes its base URL, sends its normal request and adds a task signature such as:

• riftcms-mcp-agent@v1
• riftcms-mutation@v1
• unionnexus-5w@v1
• unionnexus-legal-citation@v1

The signature identifies the kind of work being performed. Clerecy uses it to apply the correct routing policy, validation rules, evaluation rubric, exploration budget and data-retention limits.

For every request, Clerecy can:

1. Classify the work and select an appropriate model.
2. Route simple tasks to a faster or less expensive tier.
3. Validate the response against deterministic requirements.
4. Escalate to a stronger model when the first result fails.
5. Record which model actually served the request.
6. Measure cost, latency and prompt-cache behavior.
7. Connect the result to the human reviewer’s eventual decision.
8. Use that evidence to improve future model selection.

This is already production infrastructure. RIFT CMS and UnionNexus route live AI traffic through Clerecy AI Router. The dashboard currently tracks more than 700 routed requests across 15 task signatures, 11 models and four providers.

Verified cost-aware routing

Cheap routing is easy to claim and surprisingly difficult to prove.

Clerecy AI Router supports speculative cascades: begin with a low-cost model, validate the output and escalate only when necessary. Applications can attach deterministic rules such as:

• Required words or tokens
• Forbidden language
• Required JSON schema keys
• Task-specific output constraints

If the inexpensive model violates a rule, its answer is rejected before reaching the application and the request moves to a stronger model.

The response records the complete chain. If a request began on a free tier but ultimately required a flagship model, Clerecy AI Router identifies the earlier model in cascadedFrom, attributes the final response to the model that actually served it and reports zero savings for that request.

We verified this from outside the router using adversarial client probes. One probe deliberately required a result that the free tier would fail. The production response showed the failed attempt, escalation and final model attribution.

Exploration is controlled per task signature. It can be enabled for low-risk work and locked to 0% for high-stakes operations such as governed CMS mutations. Learning is never allowed to silently degrade sensitive user-facing work.

Human review becomes the learning signal

In regulated workflows, AI output does not become official merely because a model generated it.

RIFT CMS and UnionNexus already require a person to review the result. Clerecy captures the outcome of that review against the exact AI call:

• Approved: the output was useful as written.
• Approved with edits: the corrected text and change size become supervised evidence.
• Rejected: the original output becomes a negative example.

This creates task-specific evidence that no public benchmark can provide. It tells us whether a model successfully performed our work, under our rules, for the people who actually use the result.

Gemini 3.7 Flash evaluates sampled traffic asynchronously against a rubric attached to each task signature. It produces standardized scores and failure codes without adding latency to the original application request.

The strongest examples enter a nomination queue where a human can promote them into a frozen golden exam with one click.

Separate training data from the exam

Automated evaluation creates a subtle problem: if an AI judge selects the examples used to evaluate models, the system may reward outputs that please the judge rather than outputs that help users.

Two judges agreeing with each other does not necessarily establish correctness. They may simply share the same bias.

Clerecy AI Router therefore maintains two deliberately separate datasets:

Distillation corpus

A large, continuously growing collection of deduplicated, high-scoring examples. It is built asynchronously from production traffic and can be exported as training-ready JSONL.

Human corrections can also produce preference pairs for future DPO training.

Frozen golden exam

A small benchmark built from real human review decisions. Examples are promoted deliberately and then frozen. They are used to compare new model releases and gate distilled specialist models.

A model is never promoted merely because it performs well on examples selected by its own evaluator.

This separation gives us both scale and trustworthy evaluation.

Autonomous drift detection

Foundation-model providers can change behavior behind a stable model name.

Clerecy AI Router runs daily drift canaries to detect those changes before users encounter them. Each canary sends a pinned task through a production route and compares the result with its established baseline.

Current probes cover workflows including:

• RIFT CMS design-token mutations
• Federal statutory citations
• UnionNexus grievance intake guardrails
• Brand and tone consistency

Gemini grades the results off the request path. Clerecy AI Router reports score changes and new failure codes and operates under a hard daily budget cap.

This turns model monitoring from a manual spot check into a background operational process.

Telemetry

Every call creates a durable record.

Clerecy AI Router stores a compact operational index in Firestore and writes hourly compressed JSONL archives to Cloud Storage. Larger payloads are offloaded to separate blobs under configurable per-signature limits.

Capture happens outside the critical request path. If archival storage is temporarily unavailable, applications can continue routing requests.

The dashboard reports:

• The model that actually handled each request
• Input and output token usage
• Prompt-cache behavior
• Actual cost
• A named flagship comparison baseline
• Counterfactual savings
• Median and p95 latency
• Validation and escalation history
• Human-review outcomes
• Evaluation scores and failure codes

Savings are never presented as an unexplained percentage. They are calculated against a named model at its published price for those exact requests.

Dispatch is also fail-closed. If provider credentials or upstream services fail, Clerecy returns an explicit error. It never fabricates an answer to make the route appear successful.

Multi-model consensus

Some tasks justify more than one candidate answer.

Clerecy AI Router includes a tournament workflow that dispatches a request to multiple models in parallel, evaluates the candidates and catalogs the winner. The dashboard exposes the competing models, their provider and cost class, and the judging result.

This lets an organization establish evidence for which models perform well before changing a production route.

It also prevents “best model” from becoming a permanent architectural assumption. A new model can earn production traffic by beating the incumbent on the organization’s own work.

How we built it

Clerecy AI Router runs on Google Cloud:

• Cloud Run hosts the OpenAI-compatible gateway, background services and dashboard.
• Firestore stores the operational index and task configuration.
• Cloud Storage stores compressed archives and larger captured payloads.
• Cloud Build and Artifact Registry provide the deployment pipeline.
• Gemini 3.7 Flash performs asynchronous evaluation and drift grading.
• Google GenAI SDK provides Gemini integration.
• Node.js and TypeScript power the backend.
• React powers the control center.

The architecture separates routing from capture, evaluation and maintenance work. Deep evaluation does not delay the application that made the request, and failure in an analytical subsystem does not create a fabricated response.

Built by collaborating agents

The development process mirrored the product’s governance model.

We began with a one-shot prototype in Google AI Studio and deployed it to Cloud Run. Development then moved into Google Antigravity, where Gemini built the router and its cloud infrastructure.

A second coding agent, Claude, worked inside the RIFT CMS and Union Nexus repository as the client-side architect. It implemented the integration, tested the router from the caller’s side and challenged the API contract.

The two agents found problems that neither side could see alone.

When client probes showed that the classifier selected a cheap model while the flagship silently served the request, the router-side agent traced the problem through three disconnected execution layers and fixed the production path.

A human reviewed and approved the work throughout—exactly the model Clerecy supports for its own users.

Challenges

The first production traffic exposed a streaming race

Within the first hour, long-running agent jobs began failing with intermittent 503 responses.

Because both sides of the connection retained correlated records, we traced the failure to a write-after-end race in the server-sent-events handler. The final usage frame could be written after the response had closed.

We fixed the race and added transient-failure retry behavior to the client applications.

Routing intentions and execution diverged

Our client-side probes caught the classifier choosing a free-tier model while the flagship handled the call. The router’s internal logs made the route look successful and credited savings that had not occurred.

We threaded actualModelUsed through execution, validation, attribution and cost calculation so every layer now describes what genuinely served the request.

This became a core design rule: verify routing claims from the caller’s side of the wire.

Agents confidently generated stale identifiers

Coding agents repeatedly produced plausible but outdated model names in UI labels and health endpoints. Prompting did not solve the underlying problem.

We changed the architecture so model identifiers and labels must come from the live dispatch catalog. Hand-typed model names are no longer treated as a source of truth.

Anything an agent can type from memory will eventually become stale. Operational truth must come from configuration or live APIs.

Evaluation risked becoming circular

An AI judge can help score examples, but it should not quietly define the benchmark it will later use.

We anchored golden-exam promotion in real human review outcomes and kept those exams separate from the automated training corpus.

Accomplishments

• Two production applications route AI work through Clerecy.
• More than 700 calls have passed through the live gateway.
• Fifteen task signatures represent distinct regulated workflows.
• Deterministic validation and escalation have been tested adversarially from the client.
• Human decisions can be connected to exact calls end to end.
• Drift canaries run against production models under a fixed budget.
• Golden-exam candidates are grounded in human-approved work.
• Training and evaluation datasets remain deliberately separated.
• Cost and savings claims reflect the model that actually served each request.
• The system fails explicitly rather than fabricating successful output.

What we learned

Verify from the outside

A control plane’s logs describe what it intended to do. Client-side evidence shows what actually happened.

Human review is valuable proprietary data

Regulated organizations already spend time reviewing AI output. Capturing those decisions requires little additional labor and produces evaluation evidence that cannot be purchased from a benchmark provider.

Full capture makes failures useful

The same records that helped us resolve a streaming failure now power model comparison, drift detection and future distillation.

Agent memory is not operational truth

Model names, schemas and supported capabilities should come from live catalogs and closed vocabularies—not an agent’s recollection.

Honest telemetry changes engineering behavior

Once actual model identity, escalations and zero-savings outcomes are visible, it becomes much harder to hide routing mistakes behind aggregate percentages.

What’s next

Our next milestone is to freeze the first production-derived golden exams.

When a provider releases a new model, Clerecy will be able to produce a task-specific comparative scorecard within hours. The model will earn traffic only where it demonstrates better cost, latency or quality on the organization’s own frozen exam.

We will then distill the highest-volume task into a small open-weight specialist using human-approved outputs and corrected examples. The specialist will be promoted only if it beats the incumbent on an exam kept separate from its training data.

Finally, we will connect the rest of Clerecy’s products. Every new application will contribute its own task signatures, review decisions and operational evidence.

The long-term thesis is simple:

If a person already reviewed an AI answer before it became official, that work should make the next answer better.

Built With

• ai-studio
• antigravity
• cloud-build
• cloud-firestore
• cloud-storage
• express.js
• gemini3.7
• google-cloud-run
• react-19
• tailwind
• typescript
• vite