InterviewKit

Evidence-backed project interviews that reveal how developers reason, direct Codex, test, and ship, not just whether their final code passes.

Comment

Inspiration

LeetCode-style interviews no longer reflect how most software is built. They test whether a candidate can solve an isolated algorithm problem under artificial constraints, but reveal very little about how that person designs systems, weighs tradeoffs, tests assumptions, or ships maintainable software.

AI has made this disconnect even clearer. Employers need engineers who can use tools like Codex to build software effectively. That means more than generating code. Engineers must know how to guide AI, spot risks, verify its work, and take responsibility for the final result.

Traditional take-home assignments have a similar problem. A polished repository does not show who made the important decisions or whether the candidate understood and verified the implementation.

I built InterviewKit to make AI-assisted engineering visible and assessable. Instead of banning AI or looking only at the finished product, InterviewKit evaluates how a candidate works: how they frame the problem, direct Codex, make tradeoffs, test the implementation, and decide whether something is ready to ship.

What it does

InterviewKit turns a real project task into an evidence-backed technical interview inside Codex:

  • An employer signs in, creates a role, and generates a single-use invitation for a candidate.
  • The candidate installs the InterviewKit plugin, claims the invitation, and receives a scoped starter repository in a dedicated Codex task.
  • Before implementation begins, the candidate explains their approach, identifies an important risk or tradeoff, and describes how they plan to verify the work.
  • Codex serves as a neutral interviewer and a constrained implementation partner. It follows decisions made by the candidate without coaching them toward an expected answer.
  • Lifecycle hooks collect privacy-minimized evidence, including phase events, tool categories, aggregate prompt size, and whether bulk material was imported. Raw prompts, tool inputs, tool outputs, and unrelated Codex history are not stored.
  • After reviewing a clear disclosure and choosing to submit, the candidate authorizes InterviewKit to package and upload the project for evaluation.
  • Hidden tests and type-checking run without network access, under fixed resource limits, against a read-only copy of the submission.
  • The employer receives a report covering correctness, architecture, Codex proficiency, verification, quality and security, and communication.

The employer gets one continuous workflow:

Role → invitation → candidate attempt → isolated evaluation → employer report

The candidate gets to work in the same kind of environment used for modern software development: Codex and a real repository.

Why it is different

  • It measures the working process, not just the output. InterviewKit evaluates how the candidate directs the work, tests it, and reflects on the result.
  • It makes AI use part of the interview. Instead of pretending AI-assisted development does not exist, it tests whether candidates can use it responsibly.
  • It separates coaching from execution. Codex can implement the candidate’s decisions, but it cannot provide missing approaches, risks, tests, diagnoses, or fixes.
  • It minimizes data collection by design. InterviewKit does not need raw candidate conversations, unrelated Codex history, or OpenAI credentials.
  • It treats candidate code as untrusted. Projects run only in disposable evaluation environments with restricted resources and no network access.

How I built it

InterviewKit is a working, end-to-end hosted pilot rather than a prompt demo.

A Go application provides the employer interface and candidate APIs, exposes an alternate MCP interface, stores workflow state in SQLite, and runs an embedded evaluation queue worker. The Codex plugin includes a skill, lifecycle hooks, MCP configuration, and a signed helper. That helper claims invitations, creates assessment tasks, captures limited evidence, and packages submissions.

In production, the application creates a temporary Fly Machine for each evaluation job. The machine has no application secrets or persistent volume. It downloads only the active artifact using a random, single-use token, drops privileges, disables networking for candidate commands, adds server-owned hidden tests, runs Vitest and TypeScript checks, returns a size-limited result, and then destroys itself.

Architecture

InterviewKit Architecture

How I used Codex and GPT-5.6

GPT-5.6 in Codex was my main engineering collaborator throughout Build Week. I used it for product design, implementation, adversarial review, testing, deployment, and documentation. I remained responsible for the product direction and the decisions around security and trust boundaries.

  • Product design: Codex helped me turn the initial idea into a staged workflow for employers and candidates. It also helped identify where the product needed explicit consent, neutral interviewer behavior, and recovery paths.
  • Implementation: Codex accelerated development across the Go web application, SQLite queue, plugin skill and hooks, helper binary, MCP server, isolated evaluator, templates, tests, and Fly deployment configuration.
  • Verification: Codex ran and expanded package-level tests, reviewed state transitions and security boundaries, and verified that candidate code never runs inside the persistent application process.
  • Iteration: During review, I found a failure mode where the agent could begin implementing before receiving every candidate-owned planning element. I decided to add an atomic planning gate. The workflow now requires an approach, a risk or tradeoff, and a verification plan before tools or edits can be used.
  • Release engineering: Codex helped document and verify Developer ID signing and notarization, Fly deployment, queue recovery, plugin cache-busting, and production smoke tests.

The core product decisions were mine. I chose to evaluate how candidates direct AI rather than ban it, keep Codex neutral during assessments, discard raw prompt content, require explicit authorization before uploading, isolate candidate code, and describe pilot heuristics honestly instead of claiming to detect authorship.

Challenges I ran into

Preserving candidate agency

The hardest prompt and state-machine problem was allowing Codex to carry out the candidate’s decisions without becoming an answer key. I used phase gates, neutral follow-up questions, and candidate-originated tests to make that separation clear.

Capturing evidence without creating surveillance

InterviewKit needed enough evidence to produce a useful employer report without collecting raw prompts, commands, attachments, unrelated tasks, or OpenAI credentials.

Running untrusted submissions safely

The evaluator needed to keep candidate code away from the application database, secrets, and network while still returning useful hidden-test and type-checking results.

Shipping a real plugin

The candidate helper needed Apple Developer ID signing, a hardened runtime, notarization, stable packaging, and careful cache and version handling. Getting the source code to work on my machine was only one part of the job.

Accomplishments I am proud of

  • Built a working hosted flow from employer login and invitation to candidate submission and employer report.
  • Created a complete Codex plugin with a skill, lifecycle hooks, a helper binary, alternate MCP tools, marketplace packaging, and a downloadable production bundle.
  • Shipped an Apple Silicon helper that is signed with a Developer ID and notarized by Apple.
  • Designed privacy-minimized collaboration evidence that excludes raw prompt text, tool contents, command text, attachment contents, and unrelated Codex history.
  • Implemented disposable, network-isolated evaluation with hidden tests, frozen dependencies, read-only submission inputs, resource limits, and no inherited application secrets.
  • Built a repository test suite with 88 named Go tests, with every Go package passing before submission.
  • Created a complete Build Week commit history covering deployment, notarization, network flow, neutral interviewer behavior, and invitation UX.

What I learned

The most useful unit of evaluation in AI-assisted development is not the prompt or the final artifact. It is the working loop: how someone frames their intent, makes decisions, responds to evidence, and decides when the result is ready to ship.

I also learned that product integrity depends on many small details. Single-use credentials, same-origin checks, archive extraction limits, task binding, explicit upload consent, signing, notarization, and operational recovery matter just as much as the central AI idea.

Potential impact

For hiring teams, InterviewKit can bring several disconnected steps into one process: creating a realistic task, observing AI-assisted work, checking correctness, reviewing decisions, and preserving evidence for a hiring discussion.

For candidates, it offers a more job-relevant interview that recognizes AI proficiency while still requiring independent judgment. For engineering leaders, it provides a foundation for comparing how people work with AI, not just how quickly they produce code.

InterviewKit does not claim to identify all AI-generated text or prove authorship. The current pilot provides structured evidence and integrity signals that are meant to support human hiring judgment, not replace it.

What’s next

  • Broaden assessment coverage: Add more languages, roles, and project formats, supported by versioned rubrics and benchmarked difficulty.
  • Harden evidence integrity: Send events through an authenticated server-side channel with signatures, sequencing, and tamper detection.
  • Expand employer operations: Add organizations, multi-user identity, role-based access, invitation delivery, retention controls, and auditable review workflows.
  • Strengthen evaluator authority: Use evaluator-owned manifests and test configuration, verify expected test counts, and improve queue recovery and observability.
  • Support more candidate platforms: Create signed helpers for additional operating systems while keeping the scoped-token and explicit-consent model.

Built With

Share this project:

Updates