Content Performance Signal Agent

An evidence-first agent that reads ClickHouse engagement data, computes deterministic signals, then has Gemini explain, never invent, whether a title should renew, cancel, or reposition.

Comment

Inspiration

Content renewal decisions at streaming platforms are made by analysts manually pulling retention charts, comparing episodes, and writing up a judgment call that an executive then has to interpret secondhand. That process is slow, inconsistent between analysts, and doesn't scale across a large catalog.

We wanted to see if an agent could do more than surface the same charts faster. Could it actually perform the analytical reasoning itself, deterministically, and only use an LLM for the part LLMs are actually good at: explaining evidence in plain language, not producing it.

What it does

You give the Content Performance Signal Agent a content ID. It:

  1. Retrieves real engagement data for that title from ClickHouse (views, completion rate, episode-level retention, new vs. returning viewers).
  2. Validates the data is sufficient to reason about. If it isn't, the agent says so instead of guessing.
  3. Computes deterministic performance signals: retention decay, drop-off concentration, engagement velocity, acquisition trend.
  4. Applies a rules-based decision policy over those signals to produce a recommendation: RENEW, CANCEL, REPOSITION, RE-EDIT, PROMOTE, or INVESTIGATE.
  5. Passes the structured evidence, not raw data, to Gemini, which interprets it into a plain-language rationale for a content executive.

The recommendation and every number behind it come from code, not the model. Gemini's only job is to explain the evidence it's handed. It cannot invent a metric, override the decision policy, or claim certainty the data doesn't support.

How we built it

The core architectural decision was separating the deterministic layer from the reasoning layer as two physically distinct modules that never share responsibility. The signal engine and decision policy are plain, testable code with no model calls in them at all. Gemini only ever receives their already-computed JSON output and is constrained to a structured output schema, so it has no path to fabricating a number.

Data lives in ClickHouse and is queried through ClickHouse's official MCP server, so the retrieval step is a genuine runtime integration rather than a name-dropped dependency. The agent itself is built on Google Cloud's Agent Development Kit, deployed through the Gemini Enterprise Agent Platform.

The frontend mirrors this separation visually: evidence and reasoning are styled distinctly throughout, so anyone watching the demo can see at a glance which numbers were measured and which were interpreted.

Challenges we ran into

The biggest risk with this kind of project is ending up with a dashboard that has an LLM caption bolted on, which isn't meaningfully agentic. We addressed this by enforcing the evidence and interpretation split in code, not just in the prompt, and by making the data quality gate a hard stop rather than a suggestion. If the evidence is thin, the agent reports that honestly instead of producing a confident-sounding guess.

We also had to be deliberate about which signals actually justify an analytical database like ClickHouse rather than a simple lookup. Signals like engagement velocity and retention decay require real aggregation over time-series data, which is where ClickHouse earns its place in the architecture rather than being decorative.

Accomplishments we're proud of

A working end-to-end pipeline where every number in the final recommendation traces back to a real ClickHouse query, and where the model genuinely cannot override or fabricate the underlying evidence. The insufficient-data case was important to us: an agent that knows when not to answer confidently is a harder and more honest thing to build than one that always has an opinion.

What we learned

How much discipline it takes to keep an LLM in an advisory role instead of letting it quietly become the source of truth. It's easy to let the model compute something "just this once." Building the schema and the gate before writing the Gemini integration, rather than after, made that boundary much easier to hold.

What's next

Extending the signal set with cross-title benchmarking once we have comparable-title data, adding scheduled runs so the agent can flag titles proactively rather than only on request, and tightening the confidence scoring against a larger evaluation set of labeled outcomes.

Built With

Share this project:

Updates

Submission history