JudgeOps

• 

  JudgeOps

Inspiration

Hackathon judges, recruiters, and program reviewers often need to evaluate many submissions in a limited amount of time.

The challenge is not only reading faster. Evaluators must apply the same criteria consistently, find the strongest evidence, notice unsupported claims, and identify what still requires human verification.

A general AI chat can summarize a submission, but a summary alone does not create a reliable review process. I built JudgeOps to turn criteria and submission materials into a structured, inspectable review while keeping scoring and the final decision human.

What it does

JudgeOps evaluates one submission against criteria provided by the evaluator.

The evaluator pastes the criteria, participant submission, decision context, and optional supporting links. JudgeOps then uses Gemini to produce:

• a concise review snapshot;
• the strongest signal;
• the main concern;
• a suggested next step;
• an assessment for each criterion;
• the best evidence found for each criterion;
• remaining concerns and evidence-support labels;
• questions or checks to complete before deciding.

The evaluator assigns the scores, adds notes, records a rationale, and makes the final decision. JudgeOps does not choose winners, reject candidates, or replace the evaluator.

Supporting links remain available to the evaluator, but JudgeOps clearly states that it has not independently opened or reviewed them.

JudgeOps is designed for evaluators facing a temporary or recurring increase in review volume. The business can begin with individual paid reviews and later expand to event or team access.

View the full-size JudgeOps business model

How we built it

JudgeOps uses a Node.js and Express backend with a vanilla JavaScript frontend.

Gemini is called through the @google/genai SDK with structured JSON output. The response is organized into a review snapshot, criterion-level findings, evidence, concerns, suggested next steps, and items requiring human verification.

Best-evidence excerpts are checked server-side against the pasted submission. When an exact excerpt cannot be confirmed, JudgeOps shows a factual summary rather than presenting a fabricated quote.

Firebase Authentication provides Google sign-in. Firestore stores user-scoped reviews, saved rubrics, scores, notes, decisions, rationales, feedback, model runs, and activity events. The application is deployed on Google Cloud Run.

Evaluators can save criteria as reusable rubrics, score each criterion, track which criteria remain unscored, save the complete decision record, copy the human-facing review, and reopen previous reviews from history.

Challenges we ran into

The main challenge was defining what AI should and should not do in an evaluation workflow.

JudgeOps needed to organize evidence without turning an AI interpretation into a verdict. I had to keep evidence support separate from project quality and make it clear that “well supported” means the submission contains relevant evidence, not that the submission deserves a high score.

Another challenge was handling evidence honestly. A link included in a submission is not proof that its contents support a claim. JudgeOps therefore displays supporting links but does not claim to have reviewed them.

I also had to keep criterion-level output consistent even when users pasted loosely formatted rubrics. The interface needed to remain readable while showing an assessment, excerpt, concern, support level, score, and note for every criterion.

Finally, I implemented authentication, per-user Firestore isolation, an administrator-managed access list, and rate limiting for the deployed Gemini endpoint.

Accomplishments that we're proud of

I am proud that JudgeOps creates a complete review workflow while keeping human judgment visible and in control.

It can:

• turn free-text criteria and a submission into a structured review;
• identify the strongest signal and main concern;
• analyze the submission criterion by criterion;
• verify exact evidence excerpts against the original submission;
• distinguish supported, partially supported, and unsupported claims;
• surface questions that still require human review;
• let evaluators score, annotate, and record their own decision;
• save reusable rubrics and review history;
• preserve the original inputs, prompt, raw response, model, and execution details.

I am also proud that human control is part of the interface rather than only a disclaimer. JudgeOps leaves numeric scoring, notes, rationale, and the final decision to the evaluator.

What we learned

I learned that evaluators do not only need a summary. They need a clear boundary between:

• what the submission says;
• what the available evidence supports;
• what remains unclear;
• what the AI recommends checking;
• what the evaluator ultimately decides.

I also learned that missing evidence can be as important as positive evidence. A useful review should not only highlight strengths. It should show what cannot yet be concluded from the available materials.

Human control must also be part of the workflow. The product should make it obvious where AI analysis ends and evaluator judgment begins.

Finally, saved reviews and rubrics create value beyond one AI response. They provide a reusable record that evaluators can revisit instead of rebuilding the review from scratch.

What's next for JudgeOps

The next step is to test JudgeOps with more people who review submissions, candidates, applications, grants, or competition entries.

Future directions include:

• paid use per review;
• event or team access during high-volume evaluation periods;
• better support for recurring review cycles;
• multi-submission workflows and comparison views;
• bulk import for larger evaluation programs;
• controlled inspection of repositories, demonstrations, and supporting materials;
• clearer reporting across evaluators while preserving individual scores and rationale.

The long-term goal is not automated selection. It is to give evaluators a clearer, faster, and more consistent way to work through evidence while keeping the final decision human.

Built With

• cloudfirestore
• css
• express.js
• firebaseauth
• geminiapi
• googlecloudrun
• googlegenai
• html
• javascript
• node.js