Technical Solution Write-Up

While software engineering has shifted toward an agentic, pair-programming workflow, modern technical assessment platforms remain limited to measuring algorithmic output in isolation. They fail to evaluate how effectively an engineer guides, reviews, and collaborates with an AI coding agent. This platform addresses that gap by directly scoring the developer-agent interaction loop within an integrated, verifiable grading pipeline.

Candidates complete software engineering assignments inside an ephemeral, browser-based IDE powered by E2B sandboxes. During the session, the backend captures full LLM telemetry — including prompts, completions, and tool calls — alongside corresponding Git history. Upon submission, an automated Outcome Gate executes a hidden test suite to ensure baseline functional correctness. Subsequently, three independent, judge-tier LLM modules evaluate:

AI Collaboration Quality — focusing on verification behavior, test-running patterns, and corrective feedback
Code Quality
Test Coverage

To eliminate the inherent unreliability and hallucination risks of LLM-based grading, the system enforces a strict provenance invariant: every dimension score must resolve to explicit, verifiable evidence within the execution trace or Git log. A programmatic provenance validation layer cross-references every cited span_id and commit_sha before report generation. Any evaluation output containing untrackable or fabricated citations is instantly rejected at the composition gate, ensuring that final interviewer reports are deterministic and audit-ready.

"How We Used Kiro" Note

Challenge / Opportunity

Technical interviews haven't caught up with how engineers actually work now: with an AI coding agent as a pair-programmer. Current tools test whether a candidate can produce working code, but not whether they know how to collaborate with an AI agent (verifying its output, running tests, pushing back on wrong suggestions). We built an "HackerRank for agentic coding" platform that scores this collaboration skill directly, with every judge score backed by verifiable evidence instead of an LLM's unchecked opinion.

Our Solution

Candidates work inside a sandboxed IDE with an AI coding agent while we capture every prompt, response, and commit. Three LLM judges then score collaboration quality, code quality, and test authorship, but no score counts unless it cites a real trace span or commit hash. A provenance check rejects any judge output with fabricated evidence before it reaches the final report.

How We Used Kiro

We split the system into four vertical slices — LLM gateway, sandbox/harness, evaluation engine, orchestration — and built each as its own Kiro spec, moving requirements → design → tasks. EARS-format acceptance criteria kept scope explicit down to exact rubric dimensions, so nothing drifted during implementation.

For the evaluation engine, we defined correctness properties up front, e.g.:

"provenance check accepts iff every evidence entry resolves"

and had Kiro generate Hypothesis-based property tests alongside the implementation, giving us real evidence the judge pipeline can't leak fabricated citations.

Kiro's task dependency graph let us parallelize independent modules (bundle store, LLM client, prompt templates) instead of building strictly linearly, and checkpoint tasks caught regressions early across the ~18 correctness properties.

Impact

Spec-first, property-tested development meant our highest-risk requirement — that scores are never fabricated — was verified by tests rather than by hope.