Render Farm Triage

• 

  Render_Farm_Triage_GUI

Inspiration

Rendering is where an artist's time and money go, and wasted node-hours hit small studios and independent creators the hardest. Cutting that waste lets more people make things, so the project interested me beyond just the technical.

The problem itself came from my background in process engineering. In a prior job, I built a relational database so wafer fab engineers could understand their process flows. There were dozens of intermingled steps and a failure at one stage often traced back to a different one. Render data is convoluted in the same way. There are hundreds of failures, several unrelated causes at once, and tools that only show counts per job or per node. Separating simultaneous root causes from noise is a problem I've worked on. I wanted to see whether an agent could solve these types of challenges.

What it does

The agent reads a batch of render tasks from ClickHouse Cloud and tests each dimension (such as shot, node group, renderer version, and time) for failure concentration above base rate, checks where the same tasks succeeded before attributing a cause, writes its own SQL through the ClickHouse MCP server when the fixed tools run out, and produces an ordered re-queue plan. A person approves each step. Nothing executes on its own

How I built it

The dataset with injected faults was generated before the agent was built. The dataset includes three known faults plus a patternless residual. That gave me ground truth so I could grade the agent's answers. The agent is built on Google ADK with Gemini deciding which tests to run. It has access to six deterministic tools that do the statistics in code and to mcp-clickhouse for free-form SQL. It runs on Cloud Run, and a Next.js UI streams every tool call live and lets a person approve the plan step by step,

Challenges I ran into

I generated two faults that throw the identical out-of-memory error, so grouping by message merges them. This was difficult for the agent because the only way to separate the faults is to by cross-tab task weight against node memory. The same out-of-memory error has different fixes. I also added a set of faults with no pattern and needed the agent to indicate that and not invent one. Getting the agent to identify all of these correctly took several prompt iterations.

Accomplishments that I'm proud of

Every number in the agent's report matches ground truth. It found all three injected faults, separated the two that share an error, noticed the tasks that had already recovered on their own, and declined to explain the failures that have no pattern. The plan it produces is safe to follow in order. That is a working proof of concept for an agent assisting with render farm failures,

What I learned

ClickHouse was new to me. I've done data analytics and it's good to have another tool that's fast enough for an agent to iterate on. I also learned that designing the faults and the verification tests was as much of the work and as critical as building the agent itself. The agent got better because it was tested against known right answers that covered a range of failure types.

What's next for Render Farm Triage

Connect it to a real farm's logs instead of simulated data. Also test a fault that I deferred. The fault is a frame that renders fine on a 24 GB node but fails on a 16 GB one. The frame isn't broken and the node isn't broken. Failures like that look random until you compare frame size against node memory. That's a test of whether the approach generalizes,

Built With

• clickhouse
• clickhouse-cloud
• gemini
• google-adk
• google-cloud-run
• mcp
• mcp-clickhouse