Cognit.ai

Inspiration

Every student has an AI assistant one tab away. Universities are stuck between banning it, which doesn’t work, and allowing it, which means a passing test suite no longer tells you whether the student understands their own code.

We wanted to try the third option: let students use the kind of help a good tutor gives, then check whether the understanding survives when the help is taken away.

What it does

COGNIT is a live Python assessment for university courses. It allows bounded AI support and measures what students can do without it.

Write and run real Python in the browser. A Monaco editor runs the student’s function against example tests using Pyodide, with timeouts that kill infinite loops cleanly.

Code and understanding are scored separately. Tests are worth 70 points. A written explanation is worth 30, graded by a separate AI grader against a rubric covering the core idea, complexity and trade-offs, and edge cases. So a syntax slip costs you test points without wiping out credit for a sound idea, and a beautifully written explanation of wrong code doesn’t get full marks.

Help, with guardrails. Lookup answers Python documentation questions with citations to python.org. Hints and Plan Mode only unlock after a first attempt — hints get progressively more specific, and Plan Mode asks Socratic questions about the student’s own plan. Asking for the answer, for the hidden tests, for “the answer but call it a plan”, or attempting prompt injection all get refused.

Assisted, then independent. If a student gets there with help, we don’t just move on. They get a linked transfer problem: a different task on the same concept, with help locked. Two independent successes raise the difficulty. A failure sends them to remedial practice on the same skill.

A conceptual follow-up. After each attempt, one short question — “what happens when the list grows from 10 to 10,000 values?” — which separates memorised wording from actual understanding.

Teacher dashboard with a human in the loop. Teachers see every attempt, the AI’s rubric breakdown and its reasons, every help request including the refusals, mastery by skill, and the gap between assisted and independent performance. Any AI grade can be overridden or approved, and every review goes into an append-only audit log.

How we built it

Frontend: React, TypeScript, Vite, Monaco and Tailwind. Student code runs in a fresh Web Worker per run, terminated on timeout.

Backend: FastAPI and SQLAlchemy on SQLite, with a persistent Railway volume. One Docker image serves both the API and the built frontend.

AI: Claude, behind a single provider module with timeouts, a retry budget and call quotas. The tutor only ever sees the public question — never reference solutions, never hidden tests. The grader returns structured JSON, and anything malformed, low-confidence or suspicious goes to the teacher instead of being trusted.

Questions: vetted parametrised templates across loops, hashing and complexity, at three difficulty levels, each with an approved transfer partner. A script runs every reference solution against its own tests before a question can be loaded.

Testing: we froze the integration contract early, then built end-to-end and failure suites that run against a mocked model so they cost nothing. The failure suites cover model timeouts, rate limits, invalid JSON, out-of-range scores, prompt injection and solution leakage.

Challenges we ran into

Four people, one contract. We froze the API shapes early and the Python and TypeScript definitions drifted anyway — test IDs (ex0 vs example-1), how citations were represented, how test inputs were encoded. We fixed it by making one shared contract module the single source of truth and documenting the browser-runner format properly.

Adaptive logic vs retries. Our learner model only accepts one piece of evidence per question, which crashed retries after an overnight update. Now each attempt only previews the outcome, and the real record is written once, when the student moves on. Running untrusted code safely-ish. Python never runs on our server. In the browser, the first Pyodide download could blow past the timeout, and an infinite loop must not freeze the page. We ended up with a preloaded spare worker, separate load and execution deadlines, and a fresh worker per run. This is a prototype, not an exam-grade sandbox, and we’d rather say so.

Trusting the browser. Test results come from the client, so the scorer re-checks every reported value and its type, rejects duplicate or invented results, and flags every attempt as untrusted evidence. Keeping help useful but bounded. Our documentation lookup started too narrow and refused “how do I use the modulo operator?” — which is exactly what the first question needs. We widened it to the concepts our questions actually require while keeping the answers neutral. A key-exposure scare. An API key was disclosed during development and had to be revoked. Keys now live only in server environment variables, and our test suite fails if one ever shows up in an API response or a committed file.

Bugs we only found live. A final audit against our own test plan caught database commits happening after responses were sent, so a fast click on Next could see stale data. We shipped a fix with regression tests that fail without it. The same audit surfaced the progression gaps in our limitations below. Merging under time pressure. Parallel frontend work, branch permissions and a single-service deployment all had to land in the last few hours.

Accomplishments that we're proud of

A working end-to-end loop on one public URL: code, run, AI rubric grading, bounded help, transfer task, teacher review, audit log. Our adversarial test set: 8/8 prohibited prompts refused, 8/8 legitimate prompts answered. These are illustrative tests, not a security guarantee. AI failures degrade gracefully and never silently — no automatic points, a clear flag, and a place in the teacher’s review queue.

What we learned

A passing test suite says very little about understanding. The gap between assisted and independent performance is a far more honest signal, and it’s the thing we’d build everything else around if we started again. Guardrails have to be layered: deterministic filters, a constrained prompt, output checks, human review. We tried relying on each one alone at some point during the weekend, and none of them held. Writing the test plan before building was the single best decision we made. It’s what caught our last round of bugs.

What's next for Cognit.ai

Server-side sandboxed execution, an isolated container per run, so hidden and edge-case tests count and code points can’t be faked. Level-aware question selection, so “harder” always means harder and a struggling student never gets moved up. Calibration with real teachers: measure how closely our grader agrees with human marks on a graded sample, and publish the numbers alongside the sample sizes. Authentication, roles and LMS integration for Moodle, Canvas and Blackboard, so teachers can run real cohorts with consent. Teacher authoring tools — write a question once, generate validated variants, approve them, pair the transfer tasks. Richer Plan Mode: multi-turn Socratic dialogue that tracks how a student’s plan changes. Cohort analytics showing which concepts transfer poorly across a whole class, so it informs teaching and not just grading. Accessibility and fairness reviews, so that needing help is never what costs you marks.