Hemat

Inspiration

A team ships an LLM help-desk assistant. A month later, someone improves answer relevance by asking the large model to rerank the retrieved passages, one call per passage. Answers get a little better. CI is green, tests pass, reviewers approve. Cost and carbon per question triple, and nobody notices until the cloud bill arrives.

I have watched this happen on an agentic AI product I work on. Every merge request was reviewed and every pipeline was green, yet our model costs kept creeping up, and we only found out when the bill came in. No single change looked expensive in review. The cost lived in how many model calls each change added at runtime, and nothing in our review process measured that.

Existing carbon tools mostly measure CI pipelines and post tips. They do not gate a merge, do not fix anything, and miss the largest runtime cost of an AI app: inference. Past GitLab hackathon carbon projects listed carbon budgets, merge request carbon diffs and live grid intensity as future work. Hemat starts there.

Hemat means "thrifty" in Indonesian.

What it does

Hemat gives every merge request in an LLM-powered app a carbon and cost budget.

1. Measure. On every merge request that touches the app, a CI job replays a fixed load profile of real questions against the change and its baseline, and writes a carbon report: SCI in gCO2e and dollars per 1,000 questions, with every term, the tokens per model and a noise margin.
2. Gate. A deterministic gate checks code quality, security, answer quality on a hidden holdout set, performance, then the carbon budget. It posts the result on the merge request, step by step.
3. Fix. When the carbon budget fails, a dispatcher job reads the report and wakes exactly one GitLab Duo agent. The Right-sizer reads the report, finds where the extra tokens come from, and opens a fix merge request against the developer's branch. The agent never judges its own fix: the same pipeline re-measures it, and the developer decides whether to merge.
4. Release. A person approves the production release in one manual job. A Duo flow writes the release narrative, and Cloud Run rolls the new version out as a 10% canary with automatic rollback.
5. Watch. Production drift checks open issues when production SCI or error rate leaves the budget. Every kept fix lowers the budget ceiling, so the bar only moves down.

The Scheduler agent applies the same loop to scheduled jobs and moves them into the cleanest hours of the Singapore grid.

Tanya is the reference app: a RAG help-desk bot on Google Cloud that answers questions about Hemat from its own documentation, with citations.

The headline run

The scripted regression (!55) added LLM reranking to Tanya:

The gate rejected !55 at the carbon budget. The dispatcher woke the Right-sizer, which cited the report fields it used and opened a one-line fix in about 75 seconds for 2.2 GitLab Credits. The fix saved 888.4 gCO2e and USD 4.02 per 1,000 questions. Hemat also counts its own footprint: this fix cost 235.2 gCO2e in CI jobs, replays and the agent run, and pays for itself after about 265 questions.

How we built it

• GitLab Duo Agent Platform. Custom flows with inline agents, created as AI Catalog items and mirrored in the repository under .gitlab/duo/flows/. Flows are started from CI through the Duo workflows API by a dispatcher job that decides from changed paths and report thresholds, so an agent runs only when a merge request needs one.
• GitLab CI/CD. The meter, the holdout eval, the gate, the dispatcher, drift checks, the canary release with a manual approval, and an SCI dashboard on GitLab Pages. Every merge request in the project went through it.
• Google Cloud. Tanya runs on Cloud Run with Cloud SQL Postgres and pgvector, and calls Gemini (gemini-3.5-flash-lite, gemini-3.5-flash, gemini-embedding-001) on Vertex AI. CI authenticates through Workload Identity Federation, with no service account keys. Every resource is created by an idempotent script in infra/bootstrap/.
• Measurement. SCI follows ISO/IEC 21031:2024:

$$ SCI = \frac{(E \times I) + M}{R} $$

E is inference energy from token counts plus compute energy, I is live grid intensity for Singapore from Electricity Maps, M is embodied carbon, and R is 1,000 questions answered. METHODOLOGY.md states every coefficient, its source and its uncertainty range, and marks each number as measured or estimated.

Challenges we ran into

• Flows are not repository files. Duo flows live in the AI Catalog, triggers cannot filter on changed paths, and there is no "new commits pushed" event. We built a dispatcher CI job that starts flows through the API, so the agents wake on our rules and not on every merge request.
• Noise. LLM latency and caching make every replay a little different. The gate uses repeated replays and a noise margin of two standard deviations, so it does not fail a change for random variation.
• Model access. Vertex AI was restricted on our project for two days, then the Gemini API's prepaid credits ran out. We made the backend a single setting, priced each call by the backend that served it, and kept baselines per backend.
• Rate limits. A live demo run made about 3,900 large-model calls. Vertex throttled the bursts and returned 504s, so we added timeouts and retries to the eval judge and recorded the failed runs honestly instead of hiding them.
• Honest numbers. Google publishes no energy per token per model, so the large-to-small energy ratio uses their price ratio as a proxy, stated as a proxy with a range.

What we learned

• Carbon and cost move together for LLM apps. Fewer large-model tokens lower both, so the case holds even for teams that do not prioritize sustainability.
• Agents are most useful with a narrow job, a fixed-format input and a judge that is not themselves. The Right-sizer reads one report, writes one fix, and the pipeline decides.
• Answer quality has to be a gate, not an afterthought. A cheaper app that answers worse is not a fix, so the holdout check runs before the carbon budget.
• Counting the tool's own footprint keeps it honest. Every fix shows its break-even point.

What's next

• Run Hemat on a second app to prove it generalizes beyond Tanya.
• More agents for the other SCI levers: hardware efficiency and caching.
• Multi-region scheduling, so flexible work can move to a cleaner grid, not only a cleaner hour.

Built With

• cloud-run
• docker
• fastapi
• gemini
• gitlab
• google-cloud
• mcp
• postgresql
• python
• vertex-ai