Assessment Crash Test

• 

  The assessment moves from vulnerable to evidence checkpoints added, while deterministic grading remains intact.

• 

  GPT-5.6 solves all 12 cases, yet the task requires 0/3 student decisions—correctness alone is not evidence of understanding.

• 

  GPT-5.6 creates a three-step checkpoint: predict before AI, judge a plausible error, then repair the reasoning on a transfer case.

• 

  Not a fixed demo: edit the assessment, cases, and misconception, then rerun the same crash-test pipeline.

• 

  Model metadata, latency, per-case grades, and rule checks make every result inspectable rather than merely trusted.

• 

  Unit-test the assignment before students take it, revealing whether correct answers actually capture observable reasoning.

• 

  Teachers edit the objective, misconception, instructions, answer type, and deterministic test cases for the current task.

Inspiration

Generative AI has created a difficult problem for educators: an assignment can still produce a correct answer while revealing almost nothing about whether the student understood the material.

Most responses to this problem focus on detecting AI use or banning AI entirely. We wanted to approach it differently.

Instead of testing the student after submission, Assessment Crash Test tests the assignment before students receive it.

The idea came from software engineering. Developers use unit tests and red teams to expose failures before software reaches users. We asked: what would happen if teachers could red-team an assessment in the same way?

What it does

Assessment Crash Test runs a structured assignment through an AI-assisted student simulation and checks whether the assignment captures observable evidence of student reasoning.

The current vertical slice focuses on editable first-year Python assessments.

The workflow is:

1. A teacher loads or edits an assessment, including its learning objective, target misconception, instructions, answer type, and test cases.
2. GPT-5.6 attempts the assessment as an AI-assisted student.
3. A deterministic evaluator grades the structured answers against the assessment’s answer key.
4. The interface shows whether the AI can pass the assignment and how many meaningful student decisions the original task requires.
5. GPT-5.6 generates a task-specific evidence checkpoint with three stages:

• a prediction committed before AI assistance;
• acceptance or rejection of a plausible but incorrect AI suggestion;
• a repair or transfer task that requires applying the reasoning in a new case.
  1. Deterministic rules verify that all three stages are present.

The key result is not simply whether AI can solve the task. It is whether the assignment makes student decisions visible.

For example, the demo assessment can be solved correctly across all 12 cases while requiring zero observable decisions. Assessment Crash Test identifies that gap and redesigns the task to require three verifiable checkpoints.

How we built it

The project is built as a Next.js application with TypeScript, React, Zod, Vitest, and the OpenAI Responses API.

The system separates generative and deterministic responsibilities:

• GPT-5.6 attacks the assessment and proposes evidence checkpoints.
• Deterministic code grades the answers and verifies the checkpoint structure.
• The expected answers are excluded from the model prompt used for the attack.
• API credentials remain server-side.
• The interface records technical evidence including the requested model, execution mode, response ID, latency, per-case grades, and rule checks.

The assessment editor is schema-driven rather than hard-coded to a single example. Teachers can change the objective, misconception, answer type, and cases, or begin with a blank assessment.

Codex helped implement the end-to-end vertical slice, including the generalized assessment schema, API route, deterministic evaluator, responsive interface, tests, and failure handling.

The most important human product decision was to avoid claiming that the system can prove permanent understanding. Instead, the product makes a narrower and more defensible claim: it can redesign an assessment so that specific student decisions become observable and verifiable.

Challenges

Separating AI judgment from reliable verification

It would have been easy to ask the model to both generate and grade everything. That would make the demo simpler, but the result would be difficult to trust.

We instead designed the system so that GPT-5.6 performs the open-ended work, while deterministic code handles grading and structural verification.

Preventing answer leakage

The AI attack must solve the assessment without receiving the expected answers. Keeping the answer key available to the evaluator but out of the model prompt required a clear boundary between the assessment data, model input, and grading logic.

Making the product general without overbuilding

The first prototype was closely tied to one Python problem. We refactored it into an editable structured assessment format while deliberately avoiding a generic learning-management platform.

The goal was one complete, understandable workflow rather than a large set of unfinished features.

Handling live model failures honestly

A live API call can fail because of configuration, connectivity, or rate limits. The application supports a recorded Golden fallback, but it is visibly labeled so that users and judges can distinguish a live GPT-5.6 execution from a recorded result.

What we learned

The central lesson was that AI-resilient assessment is not primarily about making questions harder for AI.

A harder question may still be answerable by a stronger model. A more useful design asks students to make decisions that are tied to the learning objective and can be inspected later.

We also learned that generative models and deterministic systems are strongest when they have different responsibilities. GPT-5.6 is effective at finding weaknesses and proposing task-specific interventions. Deterministic evaluation provides the consistency needed to make those results credible.

Finally, we learned that the most important output of an assessment may not be the final answer. It may be the sequence of predictions, judgments, and repairs that produced it.

What’s next

The current version is intentionally limited to structured introductory programming assessments.

Future work could extend the same crash-test approach to written explanations, data-analysis tasks, laboratory work, and LMS integrations. The larger goal is to give educators a practical way to improve assessments without banning the tools students will increasingly use.

Assessment Crash Test: Red-team assignments before students take them.

Built With

• ai-red-teaming
• assessment
• codex
• deterministic-evaluation
• edtech
• education
• gpt-5.6
• next.js
• openai-responses-api
• react
• typescript
• vercel
• vitest
• zod