ccLoad

One reliable AI API gateway for Claude Code, Codex, Gemini, and OpenAI—with smart routing, automatic failover, protocol conversion, live observability, and cost control.

Comment

Inspiration

AI coding agents are becoming part of the daily development loop, but the infrastructure behind them is still fragile. Claude Code, Codex, Gemini, and OpenAI-compatible tools use different wire protocols, authentication schemes, models, keys, quotas, and upstream URLs. A single expired key, rate limit, overloaded model, or broken streaming connection can stop an otherwise productive workflow.

The usual fix is a pile of client-side scripts and manual provider switching. That moves the complexity everywhere and still cannot answer the important question: what exactly failed—the key, the model, the channel, or the URL?

We built ccLoad to make that failure domain explicit and keep AI workflows running through one reliable endpoint.

What it does

ccLoad is a multi-protocol AI API gateway for Claude Code, Codex, Gemini, and OpenAI clients. Applications connect once; ccLoad manages the upstream complexity.

Its core capabilities are:

  • Smart routing: priority-based selection with smooth weighted round-robin across equivalent channels.
  • Automatic failover: failures are classified at Key, model, channel, URL, or client scope, so ccLoad skips only the broken component.
  • Model-aware cooldown: a failing model can cool down without disabling every other model on the same provider.
  • Multi-URL scheduling: upstream URLs are selected using observed latency and health.
  • Protocol conversion: all 12 directed conversion paths between Anthropic, OpenAI, Gemini, and Codex protocol families, including streaming and non-streaming responses.
  • Soft-error detection: HTTP 200 responses containing real JSON, text, or SSE errors enter the same failover path as ordinary upstream failures.
  • Live observability: active requests, logs, token usage, time to first byte, cost, and upstream details are visible in the built-in dashboard.
  • Cost and access control: per-channel and per-token limits, model restrictions, channel allow/deny rules, RPM limits, and concurrency caps.

How it works

A request passes through authentication, route dispatch, channel selection, protocol transformation, URL selection, and upstream forwarding. Native requests bypass conversion; cross-protocol requests pass through a central protocol registry.

The most important design decision is the error classifier. A 401 or 403 cools the current key. A model-specific failure cools the actual upstream model. A network or URL failure switches the channel or endpoint. A genuine client error returns immediately instead of wasting retries. This precise isolation prevents one bad model or key from taking healthy capacity offline.

Streaming responses are normalized at the gateway boundary, including tool calls, reasoning data, usage accounting, signatures, and SSE framing.

How we built it

ccLoad is written in Go using Gin for HTTP routing and Sonic for fast JSON processing. The service is organized around small, explicit components:

  • channel, key, model, and URL selectors;
  • a protocol registry with native pass-through and local transformation;
  • a request attempt loop with scoped cooldown and failover;
  • SQLite, MySQL, and PostgreSQL storage backends;
  • an embedded web dashboard for configuration, logs, metrics, and testing;
  • Docker and multi-architecture release builds.

The default deployment is a single binary with embedded SQLite, so users can start locally without operating a separate database. Larger deployments can use MySQL or PostgreSQL.

Challenges we ran into

The hardest problem was not forwarding HTTP. It was preserving semantics across incompatible protocols.

Tool calls, reasoning blocks, signatures, usage fields, stop reasons, and streaming event order do not map cleanly between providers. A converter can produce valid JSON and still be wrong on the wire. We therefore placed protocol behavior behind one registry boundary and treated JSON/SSE framing as a public contract.

The second challenge was error scope. Providers frequently return ambiguous status codes, model-specific failures as generic 5xx responses, or errors inside successful HTTP 200 streams. Retrying everything creates storms; cooling everything destroys availability. The classifier and cooldown hierarchy were built to make the smallest correct failure decision.

The third challenge was concurrency. Selection, cooldown, usage accounting, live request tracking, and streaming cancellation all share state. We kept the design explicit, race-safe, and testable instead of hiding behavior behind complex abstractions.

Accomplishments that we're proud of

  • One endpoint works with four major AI API protocol families.
  • Protocol transformation covers requests plus streaming and non-streaming responses.
  • Failover isolates bad keys, models, channels, and URLs instead of applying a blunt global cooldown.
  • HTTP 200 soft errors and SSE rate-limit events are handled correctly.
  • Operators get real-time visibility and accurate cost controls without external monitoring infrastructure.
  • The project remains easy to deploy as a single binary or Docker container.

What we learned

Reliable AI infrastructure starts with semantics, not retries. You must know what failed before deciding what to retry or cool down.

We also learned that protocol compatibility is defined by edge cases: streaming order, tool-call deltas, reasoning metadata, cancellation, and usage accounting. Those details matter more than a happy-path JSON example.

Finally, operational simplicity is a feature. A boring single binary with clear boundaries is more useful than a clever distributed system that users cannot debug.

What's next

Next we plan to expand protocol conformance coverage, publish reproducible routing and transformation benchmarks, improve multi-instance coordination, and add more provider adapters without weakening the central protocol boundary.

The goal stays simple: let developers choose the best available AI model without rewriting clients or babysitting upstream failures.

Built With

Share this project:

Updates