Coshee

Coshee grades a folder of student work against the teacher's own rubric, drafts feedback for every student, and emails it — but only after the human teacher reads it and types APPROVED.

Comment

Coshee: Agentic Grading and Feedback for Teachers

This document outlines the details of Coshee for the hackathon submission.

Inspiration

Grading is slow, but not because reading student work is slow. It is slow because it is forty small clerical chores wearing one name: open the file, work out whose it is, locate the rubric, read and evaluate, score, write something specific and kind, copy the feedback, draft an email, log it in a spreadsheet—then repeat that thirty times.

Coshee was inspired by the creator's wife, a dedicated teacher who spent her Sunday nights bogged down by this repetitive, exhausting admin work. The project is named after her. She is the muse, the user, and the ultimate arbiter of the system.

The goal of Coshee is simple: automate the forty minutes of clerical work wrapped around each grade, while keeping the teacher in absolute control of the grading standard and decision.

What it does

Coshee is an agentic workflow built with Google's Agent Development Kit (ADK) that turns a folder of student work into graded, reviewed, and delivered feedback. The teacher only has to say yes.

  1. Intake and Extraction: Coshee watches an assignment folder in Google Drive. It reads student submissions in almost any format—PDFs, Word documents, plain text, or even camera photos of written pages.
  2. Student Matching: It identifies who wrote each file against a roster of students. If it cannot identify a student or if there is a duplicate submission, it holds the submission for human review instead of guessing.
  3. Rubric-Based Grading: It evaluates the work against the teacher's custom rubric, justifying every point awarded or lost by citing specific quotes from the student's submission.
  4. Human-in-the-Loop Ledger: Instead of sending emails automatically, Coshee puts the draft grades and feedback in a shared Google Sheet. The teacher reads, makes any necessary adjustments, and types APPROVED in the status column.
  5. Secure Dispatch: Only when the teacher approves a row will Coshee send the feedback. It delivers a personalized email to the student with their grade and detailed notes.
  6. Summary Receipt: At the end of a run, it emails the teacher a concise digest of the actions taken (e.g., 31 processed · 29 sent · 2 held for you · an evening back).

How we built it

Coshee is implemented as a multi-agent system using Google's Agent Development Kit (ADK). The architecture consists of five specialized agents:

  • coshee_agent (Coordinator): The root agent that plans, schedules, and delegates tasks.
  • submission_agent (Per-Submission Orchestrator): Manages a single student submission end to end, delegating to the extraction and grading agents.
    • extraction_agent: Handles parsing and extracting text from various file formats. It leverages python-docx and ADK's load_artifacts mechanism to fetch file bytes and convert text-like files to plain text or pass PDFs/images natively to Gemini.
    • grading_agent: Matches the student against the class roster, parses the rubric, and scores the extracted submission while validating quotes.
  • dispatch_agent: Resolves approved feedback from the ledger and uses Google's Gmail API toolset to send emails.

Key Technical Systems:

  1. The Send Gate (SendGatePlugin): Giving an agent Google's real Gmail tool raises the risk of accidental or rogue mailings. We built a runner-wide SendGatePlugin attached to the App that intercepts calls to Gmail's send tools. The agent never composes or holds the raw message body; instead, a tool creates a secure payload in session state and returns a short handle (coshee-payload:<row id>). The plugin swaps the handle for the verified payload at tool execution time, preventing any prompt injections or rogue models from emailing students directly.
  2. Deterministic Guardrails: The grading logic is wrapped in unit-tested Python guardrails (app/guardrails.py and app/roster.py). The agents decide when to invoke these tools, but cannot alter the outcome (e.g., scoring limits, roster matching, quote verification).
  3. Headless OAuth & CLI Integration: The application implements both interactive browser OAuth consent (InstalledAppFlow locally) and platform-injected OAuth credentials for headless deployment in an Agent Runtime (Reasoning Engine) container.
  4. Mocked Offline Testability: We built a ScriptedLlm model provider (COSHEE_MODEL=fake) that reads conversations and simulates model decisions. This allowed us to run the entire multi-agent tree and call real tools offline, resulting in a test suite with 146 unit and integration tests.

Challenges we ran into

  • Base64 Payload Copying in LLMs: Originally, the grading agent passed base64-encoded email payloads to the dispatch agent to send. However, real Gemini models consistently failed to echo back long strings of base64 characters byte-perfect. This triggered security checks and caused all dispatches to fail. We resolved this by keeping payloads in session state and giving the model a payload_token handle. The SendGatePlugin then swaps the token for the real bytes on execution.
  • Real Model Quirks vs. Mocked Fakes: The ScriptedLlm mock was a perfect byte channel, which hid two bugs: a json.dumps serialization crash on binary types.Part arguments, and a bug where a real LLM decorated rubric criteria names (e.g., matching "Tesis (4)" instead of "Tesis"). When real models were deployed, these caused silent stream failures or false holds. We solved these by building custom JSON encoders and sanitizing criterion keys.
  • Agent Runtime Bucket Permissions: Deployed to Agent Runtime, our submission agent initially failed silently when calling save_artifact. The issue was that the platform's service agent lacked roles/storage.objectAdmin on the cloud bucket used for storing artifacts. We diagnosed this via Cloud Logging and resolved it by configuring proper IAM policies.

Accomplishments that we're proud of

  • Robust Security Boundary: We successfully built a secure dispatch gate that ensures no language model can approve its own work. Outbound emails are strictly gated behind a durable spreadsheet cell state.
  • Shared Execution Path: The command line (coshee intake and coshee dispatch), the ADK Web UI, and the deployed Agent Runtime run the exact same ADK agent tree. There is zero duplication between CLI scripting and cloud agent configurations.
  • Safe Roster Matching: The fuzzy name-resolution system is highly refined. It resolves names with a single distinctive token (e.g., matching "j-perez" to "Juan Pérez"), but strictly holds the grade if two students could match or if duplicate submissions are found.

What we learned

  • Scripted Mocks Aren't Enough: While mock testing is valuable for CI pipelines, integration testing with real LLMs is essential. LLM outputs are noisy, and models will struggle with tasks (like echoing base64) that a deterministic script handles easily.
  • Keep Tool Allowlists Tight: Shipped toolsets like Gmail and Drive are incredibly powerful, but exposing all of their APIs (e.g., trashing emails or deleting settings) to an LLM is a major security risk. Restricting toolsets to strict allowlists (e.g., exposing only 1 of 79 Gmail tools) makes the system secure and improves model tool-calling accuracy.
  • Durable External State: Storing the "state" of the grading workflow in a Google Sheet instead of in-memory agent sessions allows the teacher to review and approve drafts at her own pace over several days, surviving any application crashes or container restarts.

What's next for Coshee

  • Grading Calibration Metric: Establish a gold-standard dataset of teacher-graded essays to continuously test the grading agent's prompt, ensuring its scores consistently fall within the required $\pm 1$ point of the teacher's original grade.
  • Direct Google Classroom Integration: Pull student submissions directly from Google Classroom assignments and sync finalized grades straight back to the Classroom gradebook, removing the need for intermediary Drive folders.
  • Handwritten Submission Processing: Extend the extraction agent to natively leverage Gemini’s multimodal capabilities to perform OCR on handwritten homework photos.
  • Tone-Matching RAG: Build a retrieval system over the teacher's past graded feedback to automatically match her writing style, vocabulary, and grading tone when drafting new student responses.

Built With

  • agent-runtime
  • ai-guardrails
  • docker
  • gcp
  • gemini
  • gmail-api
  • google-adk
  • google-cloud
  • google-drive-api
  • google-sheets-api
  • human-in-the-loop
  • manager
  • multi-agent-systems
  • multimodal-ai
  • oauth-2.0
  • pytest
  • python
  • python-docx
  • reasoning-engine
  • secret
  • security-plugin
  • terraform
  • uv
  • vertex-ai
Share this project:

Updates

Submission history