COGNISECT

Compile the next question, not a diagnosis.

Comment

Short description

COGNISECT helps a mathematics teacher test competing formal explanations for one signed-integer error. GPT-5.6 maps observed work into a closed rule registry; deterministic code scans 625 problems and compiles one separating probe; the teacher controls both release and interpretation.

Problem

One wrong answer can fit more than one plausible procedure. If an AI system chooses one explanation too early, it can turn ambiguity into an unjustified claim about what a learner thinks. The teacher then sees a polished conclusion without a way to inspect what evidence could separate the alternatives.

COGNISECT takes the narrower problem seriously: when several formal rules still fit the observed work, which next question receives the highest rank under a frozen deterministic ordering because those rules predict different answers?

Inspiration

The project began with a boundary, not a chatbot prompt: a formal rule can explain an answer, but it cannot prove a learner's cognitive state. That suggested a different role for AI in formative assessment. Let the model perform constrained mapping, let deterministic code expose the disagreement, and keep the teacher responsible for every learner-facing and interpretive decision.

The result is an evidence workbench built around a visible transformation captured in the final production workflow: 625 problems -> 624 eligible follow-ups -> 444 deterministic separators -> rank 1 -1 - (0), with predictions -1 and 1.

What it does

The teacher starts with a provenance-cleared, educator-authored project exemplar or enters de-identified work for one signed-integer subtraction problem. GPT-5.6 maps that observation to instances from a closed, literature-grounded registry of formal procedures. It cannot write or execute a new rule.

The Counterexample Compiler then evaluates every ordered subtraction problem whose operands lie in [-12, 12]. It excludes the original problem, rejects candidates where the represented procedures agree, and applies a frozen deterministic ordering to the remaining separators. The workbench shows the complete proof path, the live separating-candidate count, exact predictions, up to five ranked finalists, and the persisted rank-one probe. In the final captured workflow, 444 eligible problems separated the two accepted procedures; frozen ordering selected -1 - (0) at rank one, with predictions -1 and 1. A counterfactual preview explains what each possible predicted response would support or weaken, while labeling those branches as predictions, not observed evidence.

Nothing reaches the learner automatically. The teacher first approves or declines the probe. An approved probe creates an opaque, one-purpose learner link that opens a minimal page with no hypotheses, teacher controls, correct answer, telemetry, or runtime navigation. The learner submits one signed integer and may add a rationale that is visible only to the authorized teacher and is excluded from deterministic evidence computation.

Exact matching updates each represented procedure as supported, weakened, or unresolved. The teacher then passes a second gate by approving, editing, rejecting, or abstaining on the final note. The saved decision is read back from Postgres beside an append-only audit. The owner can also download a canonically hashed, privacy-safe receipt that excludes the original work, rationale, capability tokens, cookies, secrets, and generated prose.

The final captured receipt has canonical SHA-256 496dc522a011a8f425c85fe1de730d2f7f9d74f9c35ff5c38e239c1bd0896767; an independent canonicalization reproduced the same value and confirmed forbidden keys were absent. The receipt also preserves nullable, content-free correlation metadata for the exact model snapshot, OpenAI Responses object ID, and provider request ID. Those owner-authorized fields remain distinct, are covered by the canonical receipt hash, and were independently verified.

The compiler proves disagreement between formalized rules. It does not prove what a learner thinks.

How we built it

COGNISECT separates generative mapping from executable authority.

GPT-5.6 Terra performs the default structured mapping and drafts a teacher-facing proposal. GPT-5.6 Sol was evaluated on the same frozen six-case harness. Both routes return strict registry instances through the OpenAI Responses API. A returned model that does not match the requested frozen route fails policy before the compiler can use its output. Provider response IDs, provider request IDs, outbound client IDs, token usage, latency, and model snapshots remain distinct in the telemetry record.

The deterministic core is Python 3.12 with a total rule interpreter and an exhaustive Counterexample Compiler. FastAPI and Pydantic expose strict contracts. SQLAlchemy, Alembic, PostgreSQL 17, and LangGraph checkpoints preserve ownership, idempotency, interrupts, resumes, audit transitions, and deletion. A Postgres fixed-window limiter enforces shared quotas using HMAC buckets instead of stored client addresses. Production readiness checks the exact Alembic head, and build identity reports a full source SHA.

The frontend is Next.js 16, React 19, and strict TypeScript. Vercel serves the interface and a signed same-origin proxy; Render hosts the Docker API and Postgres preview. Route groups isolate the teacher and learner shells without changing public URLs. The API client is generated from the checked-in OpenAPI document.

Codex was the engineering collaborator across the project. It helped turn the product boundary into executable contracts, implement the registry and compiler, build the durable workflow and complete interface, write migrations, expose the compiler proof, harden replay and concurrency, create accessibility journeys, audit dependencies, and run independent review loops. The Git history makes those stages inspectable, including the deterministic core in af43cc2, durable workflow in 2b9fd53, Responses integration in 42e5bef, first complete vertical slice in 9884be9, and the final merged release in 97cd441e91f007427ec472d816bab40eb8aa3fe3.

The human builder made the product, privacy, evidence-language, custody, design, and final claim decisions. The project was created during the Build Week submission period; dated commits document the work.

Challenges we ran into

Preserving uncertainty end to end

It was not enough to tell the model to be cautious. The architecture had to prevent the model from authoring rules, selecting probes, changing evidence, releasing learner content, or approving its own prose. That required a closed formal language, truth-table canonicalization, deterministic search, explicit state transitions, and two separate teacher interrupts.

Making the proof visible without creating a second source of truth

The 625-to-1 reveal needed to be compelling and inspectable, but every number had to come from the same backend ranking pass that selected the persisted probe. The UI consumes the proof contract and never recomputes counts or rankings in TypeScript.

Making retries safe under real concurrency

Teacher commands, learner responses, model attempts, and workflow resumes can all be replayed or arrive out of order. COGNISECT combines idempotency records, optimistic versions, row locking, append-only audit rows, recovery journals, and monotonic frontend snapshots. The previous public stress gates each accepted exactly one of 50 simultaneous learner submissions and conflicted the other 49. Across five final-SHA runs, that is five accepted and 245 conflicted submissions, with five exact replays and five disposable-workflow deletions.

Keeping evidence honest

The frozen model comparison contains six project-authored educator cases and zero learner responses. We built release tests that reject claims the evidence does not support, including educator validation, diagnostic accuracy, learning gains, classroom use, and time savings.

Accomplishments we are proud of

  • A complete teacher-to-learner-to-teacher workflow with no mock backend path or demo bypass.
  • An inspectable compiler proof that starts with all 625 bounded problems, excludes the original, exposes the live separator set, and persists deterministic rank 1.
  • A strict boundary where GPT-5.6 contributes meaningful structured mapping but holds no executable or learner-facing authority.
  • Two real human custody gates, persisted final-decision readback, and an append-only audit.
  • A privacy-safe evidence receipt with canonical SHA-256 hashing, an explicit field allowlist, and independently verified nullable model snapshot, OpenAI response ID, and provider request ID metadata.
  • Atomic, replay-safe learner response handling and distributed database-backed quotas.
  • A purpose-built learner surface that exposes none of the teacher's hypotheses, telemetry, or controls.
  • A frozen comparison with item-level model artifacts, costs, latency, exact limits, and zero learner responses misrepresented as evaluation evidence.
  • Independent task reviews, strict type and schema checks, accessibility coverage, mobile reflow, migration tests, dependency audits, secret-history scanning, and production custody gates.
  • A shared Expo SDK 57 mobile shell released as a signed Android build-2 APK and verified on iOS through TestFlight, both opening the same production service as the web app.

Built With

Share this project:

Updates