Karani

Inspiration

Most AI agents are pitched on what they can do. Karani is built around what it deliberately cannot.

If you teach, you know the night before grades are due: forty essays, and most of the work is not judging. It is hunting, rereading each paper for the sentence that backs up the feedback you already plan to give. AI could do that hunting years ago. The reason it does not is the verdict: an AI grade is indefensible to a department, and indefensible to a student who appeals. So the whole category stays banned.

The fix is not a better grader. It is an agent that does all of the labor and provably none of the judging.

What it does

Karani is an autonomous overnight agent for instructors (category: The Taskmaster). Cloud Scheduler wakes it nightly; it reads every submission in the class and builds, for each student, an evidence sheet: every finding tied to a verbatim quote from that student's own writing, located in the original document. In the morning the instructor gets a brief with what needs them, what finished, and the pattern across the class, shown in the students' own sentences.

What it cannot do is grade, and that is architecture, not a system prompt:

• The observation record has no field that could hold a score, and the schema rejects unknown fields at runtime.
• Grades live in a separate Firestore database that no pipeline identity is bound to at all. The append-only role is IAM-conditioned on the events database, because datastore.entities.create cannot be scoped below a database and the server SDK never evaluates Security Rules.
• You can watch the refusal live: the hosted docket's /boundary page makes the running service attempt a grade write under its own identity, and Google Cloud denies it in front of you. pytest -m deployed asserts the same denial as the pipeline identity on every release.

One unattended run ends in six distinct outcomes, never six labels on the same one:

The scale row is a real Cloud Run Job execution: 150 submissions, 745 findings, 2,451 events, 13.6 minutes on 15 workers, zero failures. First-pass acceptance fell from 85% to 70% at scale. We publish the drop, because a number that only improves deserves suspicion.

How we built it

Event sourcing, strictly: every step appends to a create-only Firestore log with deterministic IDs, and render(runId) is a pure fold over that log (no store, no clock, no network; a shuffled log reproduces byte-identical artifacts). Analysis is gemini-3.6-flash via Vertex AI at temperature 0, orchestrated by a Google ADK SequentialAgent (dispatcher, analyst/validator, anomaly triage) through the GenAI SDK.

Every citation passes five layers before it becomes evidence: registry membership, verbatim quote, positional identity (the context around the quote, recomputed from the frozen rendition), entailment on gemini-3.5-flash-lite, and a cross-family second read by Gemma (gemma3:4b, running locally via Ollama). No single model's judgment turns a draft into evidence. Disagreements go to the instructor and are never retried, because pressure to get past a disagreeing checker is how fabrication happens.

Infrastructure: Cloud Run Jobs (analysis fan-out), two Cloud Run services (the docket and a public arena), Firestore (two named databases), Cloud Scheduler. make demo reproduces the whole pipeline offline from 187 committed model responses, zero credentials.

Challenges we ran into

The honest list is long and it is in the repo's FINDINGS.md; four favorites:

• The model our own spec pinned did not exist. The PRD said gemini-3.5-pro. Verifying pinned IDs against the live catalog is now a preflight step.
• A separate grades collection was not a boundary. Firestore IAM does not grant below the database, so our first design let any identity that could append an event create a grade. The fix is a second named database with a conditioned binding, and it is the only part of the system where the platform enforces the boundary instead of our code.
• An injection wrapped across a line break evaded our scanner. Found by the arena's very first live test, minutes after we built it. Every committed fixture kept its payload on one line, so three weeks of green tests never noticed.
• The docket could never see the runs the deployed job wrote. Firestore does not list ancestor-only documents, so list_runs() returned empty forever. Found only by deploying and reading what the page actually served.

Accomplishments and what we learned

Deployed, measured, and adversarially reviewed: 288 tests, a measurement contract where no number is publishable unless it exists in docs/metrics.json with its method, and a run-book the CI checks so the demo cannot claim things the recorded run does not contain. The biggest lesson: almost every defect we shipped was in the narration, not the engine. Claims drift; only executed checks hold.

We think verdict-incapable agents are a pattern, not a product feature: the same pipeline, unchanged, reviews scholarship applications in the repo (see /scholarship on the docket). Evidence from the agent, judgment from the professional, and a boundary between them the agent cannot cross.

Try it, right now

• Docket (brief, replay, live boundary demo, scholarship exhibit): https://karani-docket-u42sxjnqkq-uc.a.run.app
• Arena, paste any essay and watch the real pipeline refuse to grade it: https://karani-arena-u42sxjnqkq-uc.a.run.app
• Repo (offline demo: make demo, zero credentials): https://github.com/Jeremiah-Sakuda/Karani

Built With

• cloud-run
• cloud-scheduler
• docker
• fastapi
• firestore
• gemini
• gemma
• genai-sdk
• google-adk
• ollama
• pydantic
• python
• vertex-ai