CodeReason

submission-review.png
landing.png

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of## Inspiration

Two programs can produce the same wrong answer for completely different reasons. One student may have the right structure but make an indexing mistake, while another may hardcode an example or fail before reaching the core logic.

Output-only grading misses those differences, but manually collecting enough evidence for fair partial credit takes time. We built CodeReason to explore a more responsible role for AI in programming assessment: not as an autonomous grader, but as an assistant that organizes observable evidence and leaves the final decision to an instructor.

What it does

CodeReason is an evidence-first grading workspace for Python assignments.

An instructor defines either a FUNCTION or STDIN_STDOUT execution contract, selects an output comparison policy, and creates a rubric. AI can help structure rubric criteria, but those criteria remain drafts until a person approves them.

A trusted worker executes submissions and records deterministic Primary Evidence:

test results;
execution errors;
AST findings;
static findings; and
source-code locations.

CodeReason keeps this evidence separate from AI-generated Derived Analysis. When configured with an OpenAI API key, the worker uses GPT-5.6 through the Responses API to produce structured, rubric-bound score suggestions and feedback.

Suggested deductions must cite evidence already collected by the application. The system describes only what the code shows evidence of, what a pattern suggests, or a likely misconception—never what a student privately thought or intended.

Instructors compare AI suggestions with their own scores before approving a final result. CSV export keeps AI suggestions separate and leaves final_total blank until a human approves the grade.

The reviewer workspace also supports:

evidence visibility for hidden tests;
live, fixture, unavailable, and stored-live provenance;
stale-analysis handling after input changes;
preserved human review history; and
consistency checks that raise potential review issues without changing grades.

How we built it

The reviewer interface is built with Next.js, React, TypeScript, Tailwind CSS, and TanStack Query.

A FastAPI backend validates requests and stores assignments, source revisions, execution runs, evidence, analyses, and human reviews through Pydantic, SQLAlchemy, and Alembic. PostgreSQL is the Docker Compose default, with SQLite available for local development.

Execution and AI requests are handled by a separate database-backed worker rather than the public API process.

Python 3.12 submissions run in a restricted local Docker container with backend-controlled image, command, environment, resource limits, and lifecycle. Source code is transferred with docker cp into a Docker-managed volume; students cannot configure Docker commands, mounts, images, or environment variables.

The OpenAI integration uses the Responses API and Structured Outputs validated with Pydantic. The application independently checks rubric IDs, evidence references, score bounds, execution availability, and review-required conditions. The pipeline also includes a best-effort identifier and secret redaction layer together with a manifest describing which data categories may be sent externally.

The bundled demo contains five deliberately different matrix-transformation submissions: correct, wrong-index, runtime-error, hardcoded, and missing-function implementations.

Challenges we ran into

The hardest challenge was defining a trustworthy boundary between facts and interpretation. A model-generated explanation can be useful, but it must not become evidence for its own conclusion. This led us to model Primary Evidence and Derived Analysis separately across the database, API, prompts, UI, and export flow.

Running untrusted code was another major challenge. Docker offers useful defense in depth for a local demonstration, but it is not a production-grade multi-tenant security boundary. We had to fix container arguments in backend code, disable networking, apply resource limits, avoid host bind mounts, and clean up containers and temporary files across failure paths.

Privacy, hidden-test visibility, asynchronous versioning, and human approval were also cross-cutting concerns. Each one had to survive the path from storage and worker processing to reviewer screens, student-safe feedback, and CSV export.

Finally, we wanted the demo to remain honest. CodeReason labels records as live, stored live, fixture, or unavailable so a stored demonstration is never presented as a live Docker or OpenAI result.

Accomplishments that we're proud of

We completed an end-to-end reviewer workflow rather than building only an AI prompt demo.

CodeReason can define an assignment, upload multiple submissions, execute Python code, collect several forms of deterministic evidence, queue structured AI analysis, support human score approval, compare consistency fingerprints, and export grading data.

We also built explicit failure states. If Docker or the AI provider is unavailable, CodeReason records that limitation instead of fabricating an execution result or model response.

The full Docker Compose stack and sandbox were exercised on Docker Desktop, including passing, wrong-answer, and runtime-error outcomes. Automated checks cover backend policies, frontend behavior, migrations, production builds, browser flows, and the Docker execution contract.

What we learned

The most important lesson was that responsible AI assessment depends more on system design than on prompt wording.

A useful grading assistant needs constrained inputs, deterministic evidence, versioned records, visibility rules, provenance, explicit failure states, and a real human-approval state machine. “Human in the loop” cannot be only a confirmation button after an opaque model decision.

We also learned to distinguish containment from security. Docker can reduce risk in a local demo, but production execution of hostile code requires stronger isolation, operational monitoring, and a separately administered disposable execution environment.

What's next

Our immediate priority is reliability hardening: stronger pre-transmission redaction, immutable snapshots of grading inputs, version fencing around asynchronous work, better isolation between individual tests, consistent function-result serialization, and clearer timeout classification.

After that, we plan to add authentication and role-based access control, durable cleanup reconciliation, a separately isolated execution service, and larger-cohort usability testing.

Longer-term opportunities include LMS integration, additional programming languages, configurable execution environments, and research with instructors on whether evidence-linked suggestions improve grading consistency without reducing human oversight.

CodeReason is currently a local, single-reviewer Python MVP.