Judging Copilot

One repository in. Evidence-backed verdict out.
architectural diagram

Inspiration

Hackathons can receive hundreds of submissions, but judging each one often involves the same repetitive process: opening a repository, understanding its structure, reading the documentation and code, checking whether the required technologies were actually used, assigning scores, and looking for suspicious similarities.

As a hackathon participant and builder, I started thinking about a simple question:

What if an organizer could delegate that entire first-pass investigation to an agent?

That became the idea behind Judging Copilot.

The goal was not to build an AI that simply generates a score. I wanted to build a system where an organizer gives the agent a GitHub repository and the agent can investigate the submission, decide what evidence it needs, evaluate it against a rubric, verify similarity against previous submissions, and persist the result for the organizer.

What it does

Judging Copilot is an autonomous AI agent for the first-pass verification and evaluation of hackathon submissions.

An organizer provides a GitHub repository URL. The system creates a background judging job and the agent takes over the workflow.

The agent can:

Clone the submitted repository. Inspect its directory structure. Decide which files are relevant to investigate. Read repository evidence such as documentation and configuration files. Evaluate the submission against a structured judging rubric. Produce a score with a criterion-by-criterion breakdown. Compare the submission against previous submissions using embeddings. Flag suspicious similarity or low-scoring submissions. Persist job state and final verdicts in Google Cloud Firestore. Stream progress so the organizer can see what the agent is doing.

The important distinction is that the user does not manually tell the system which files to inspect or which sequence of tools to execute.

The user delegates the task.

The ADK agent coordinates the investigation.

How we built it

The backend is built with Python and FastAPI, with Google ADK providing the agent framework.

The central component is an ADK LlmAgent running through an InMemoryRunner.

The agent has access to specialized tools for:

Repository cloning Repository inspection File reading Rubric-based evaluation Embedding-based similarity checking Firestore persistence Organizer alerts

Gemini is used for the agent's reasoning and repository evaluation, while Gemini embeddings are used for similarity detection.

The judging workflow was also changed from a completely synchronous request into a background job model. A submission creates a job, and the agent continues working while the frontend can query its persistent state.

Firestore stores both the jobs and verdicts data, allowing the frontend and backend to share the state of an evaluation.

The frontend was built as a React/Vite dashboard. It provides an organizer-oriented view of submissions, scores, flags, agent activity, and detailed verdict breakdowns.

We also added defensive handling for untrusted repository content. Repository files are treated as data rather than instructions, file access is restricted to the cloned repository, symbolic links are rejected, and repository evidence is bounded before being passed to Gemini.

Challenges we ran into

The biggest challenge was making sure the system was actually agentic, rather than putting an LlmAgent around what was essentially a normal sequential Python script.

Our first implementation could execute the judging pipeline, but the agent was effectively being told to follow a predetermined sequence.

We refactored this so the ADK agent had real tools available and could inspect the repository before deciding what evidence to use.

Another challenge was reliability.

Gemini API rate limits caused several real failures during development. Instead of hiding those failures, we added a deterministic fallback pipeline so that a temporary model failure would not leave a judging job permanently stuck.

We also discovered security issues that are easy to overlook when an AI agent is reading arbitrary GitHub repositories. Repository content can contain prompt injection attempts, malicious symbolic links, or path traversal attempts.

That forced us to treat every submitted repository as untrusted input and add containment, symlink protection, content limits, and prompt-boundary protections.

Finally, we had to design the workflow so that judging could happen as a background task rather than blocking the HTTP request. Firestore job state became the persistent source of truth for the evaluation lifecycle.

Accomplishments that we're proud of

The accomplishment I'm most proud of is that the system is not just a chatbot wrapped around a judging prompt.

A real evaluation can follow a workflow like:

GitHub repository → investigation → evidence selection → evaluation → similarity verification → persistence → alert

During testing, the ADK agent successfully used the repository inspection tools before scoring a real GitHub repository.

We also verified the background workflow end-to-end, including job creation, persistent state transitions, final verdict generation, and Firestore persistence.

Another thing we're proud of is the security work. We deliberately tested cases such as path traversal, symbolic-link escape attempts, and prompt injection inside repository content instead of assuming that repositories would always be trustworthy.

Most importantly, the project solves a problem that we actually encountered: repetitive evaluation work around hackathon submissions.

What we learned

The biggest lesson was that adding an LLM does not automatically make a system agentic.

A genuine agentic workflow needs meaningful tools, state, decisions, and the ability to use the results of one action to determine what happens next.

We also learned that reliability matters just as much as intelligence.

A model can make a good decision, but real systems also need to handle rate limits, failed repositories, malformed inputs, missing evidence, background execution, and persistence.

Another important lesson was security.

Once an agent is allowed to read arbitrary external content, that content has to be treated as hostile input. A README is not automatically trustworthy just because it is documentation.

Finally, we learned to distinguish between a hackathon MVP and production infrastructure. The current background worker is intentionally lightweight. A production version would need durable task infrastructure rather than relying on an in-process background worker.

What's next for Judging Copilot

The current version focuses on the core judging workflow, but there is a lot more that could be built on top of it.

The next step would be moving the background execution to a durable queue/worker architecture so evaluations can survive container restarts and scale across many workers.

We would also like to make the judging system configurable for different hackathons, allowing organizers to define their own rubrics, required technologies, scoring weights, and verification rules.

Other possible extensions include:

Deeper code execution and functional testing in isolated environments. More sophisticated plagiarism and similarity analysis. Evidence citations showing exactly which repository files influenced a score. Human judge review and score adjustment. Batch evaluation of hundreds of submissions. Organizer analytics and ranking dashboards. Integration with hackathon submission platforms.

The long-term vision is not to replace human judges.

It is to give them an AI investigation layer that handles the repetitive work before human judgment is required.

One repository in. Evidence-backed verdict out.