Gotcha!

Gotcha! is an AI learning game where the AI intentionally makes one subtle mistake. Your mission: find it, explain it, and level up your critical thinking.

Comment

Gotcha!

An AI that lies on purpose, so students learn to catch it.

Sign in and try it, no signup needed

Two accounts are ready. Both already have challenge history, so the dashboard, the progress charts and the PDF analysis report have real data in them.

Username Password
Account 1 judge1 gotcha2026
Account 2 judge2 gotcha2026

Open both at once, in two different browsers or one normal and one private window, and you can play a live duel against yourself. Create a duel on one, join it from the other, and watch the ready check hold the clock until both sides confirm.

Worth trying in about two minutes:

  1. Generate a challenge in any subject, then try to find the planted mistake
  2. Check the three scores on the result page. Location, diagnosis and fix are marked separately, which is the whole pedagogical point
  3. Profile, then Download report, for a PDF analysis of that account's weaknesses
  4. Answer "no error" on a few challenges. Roughly one in six genuinely has none, and claiming a fault that is not there costs you

The backend is on Render's free tier and sleeps after 15 minutes idle, so the very first request may take up to a minute. Everything after that is instant.

Read this sentence

The derivative of f(x) = x² is x³/3, so the slope at x = 3 is 9.

It is confident. It is formatted correctly. It uses the right vocabulary. It is also completely wrong: that is the integral, not the derivative. The derivative of x² is 2x, so the slope at x = 3 is 6.

If you caught it, you did something a language model cannot do for you. If you did not, you just experienced the problem this project exists to solve, in the five seconds it took to read one line.

That gap, between an answer that looks right and an answer that is right, is where Gotcha! lives.

Inspiration

I kept watching the same thing happen. A question goes into a chatbot, a confident paragraph comes out, and it gets used. Not verified. Not questioned. Used. And the answer is usually correct, which is exactly what makes the habit so hard to see forming.

The instinct in education has been to ban the tools or to detect their use. Both are losing battles. So I went looking for what researchers were actually proposing instead, and found a consistent and much more interesting answer: do not remove the AI, remove the certainty.

Four papers shaped this project.

Wazan (2026) argues that uncertainty should be a deliberate pedagogical instrument, and describes explicitly controlling AI during assessment so that it generates plausible but flawed responses rather than direct answers. The abstract states that

"uncertainty is a central pedagogical concept for stimulating students critical thinking"

Hosseini (2026) goes further and treats AI error as the curriculum rather than the defect:

"frequent errors and hallucinations, often seen as limitations, offer a unique pedagogical opportunity"

Lamberti and colleagues (2025) ran this in real classrooms, building activities where students had to

"critically evaluate the accuracy and appropriateness of GAI-generated responses"

Sonkar and colleagues (2025) asked how you would even verify that an educational AI models student thinking, and answered it by generating distractors conditioned on a specific student's own misconceptions.

Read together, these describe a system nobody had built as a product. Wazan supplies the pedagogy, Hosseini supplies the framing, Lamberti supplies the classroom evidence, and Sonkar supplies the adaptive mechanism. Gotcha! is that system.

That is what I mean by research backed. Not a citation dropped into a footer, but four specific findings that each map onto a specific mechanism in the code.

What it does

Gotcha! generates content that is deliberately wrong, and asks you to prove you noticed.

The loop.

  1. You choose a subject, a topic and a difficulty, or type a topic of your own.
  2. A language model writes a short passage containing exactly one planted error, drawn from a taxonomy of 23 error types across five subjects.
  3. The correct answer is stored server side. The browser never receives it.
  4. You highlight the wrong span, name the type of fault, and write the fix.
  5. A second, separate model call marks your answer.
  6. The ground truth is revealed, and your skill rating for that error type moves.

Marked on three axes, not one.

A single score would hide the useful information. Every answer is graded on location, diagnosis and fix, independently.

Axis Question it answers What it catches
Location Did you find the right span? Flagging the complicated looking part instead of the wrong part
Diagnosis Do you know why it is wrong? Sensing something is off without being able to name it
Fix Can you state the correction? Spotting an error without knowing how to repair it

This split is the most pedagogically valuable thing in the product. In testing, my own account scored 73 percent on locating errors and 42 percent on proposing fixes. That is a specific, actionable diagnosis: I can see the mistake but I cannot articulate the repair. A single number would have said "58 percent" and taught me nothing.

It cannot be gamed.

Once you know a mistake is coming, the obvious strategy is to flag anything that looks unusual. Three mechanisms defeat that:

  • About one in six passages contains no error at all. Claiming a fault that is not there scores zero, so the game rewards reading rather than suspicion.
  • Every user carries an Elo style rating per error type. Selection is weighted inversely to your rating, so the faults that beat you appear more often.
  • Duel scores are the average multiplied by the fraction completed, so answering two questions perfectly out of ten loses to eight answered well.

Beyond the loop.

  • Multiplayer duels with a ready check, so the clock starts only when both players are ready, on a synchronised server side timer.
  • A daily challenge, pinned per calendar date, identical for every user.
  • A downloadable PDF analysis report over any time range. It computes accuracy by subject, error type and difficulty, identifies which grading axis is costing you marks, and an LLM turns those statistics into specific guidance.
  • Global leaderboards, public profiles, activity heatmaps and streaks.

How I built it

The architecture decision that matters

ARCHITECTURE DIAGRAM

Generation and grading are separate API calls with separate prompts. The model that plants the mistake never sees your answer. The model that marks your answer never sees the generator's reasoning.

This is not an implementation detail, it is the thing that makes the score mean anything. A single call doing both would be marking its own homework, and would agree with itself. Two calls means the grader has to independently arrive at the same conclusion from the passage and your explanation alone.

The ground truth never reaching the browser matters for the same reason. If it were sent and merely hidden, the answer would be one devtools panel away.

CHALLENGE

The stack

Layer Choice Why
Frontend Next.js 16, React 19, TypeScript App Router, 15 routes, KaTeX for mathematical notation
Backend FastAPI, SQLAlchemy 2 37 endpoints, automatic OpenAPI documentation, async background jobs
Database PostgreSQL Relational integrity across users, attempts, ratings and duel state
Models Groq primary, OpenRouter fallback, Anthropic Provider abstraction with automatic failover
Reports ReportLab Server side PDF generation
Auth bcrypt and JWT Cost factor 12, token derived identity
Frontend hosting Vercel Native Next.js target, preview builds per branch
Backend hosting Render Managed PostgreSQL alongside the web service, one blueprint provisions both

Deployment is declarative. render.yaml provisions the database and the API and wires the connection string between them, and vercel.json pins the frontend build. Both platforms redeploy on push.

DATA MODEL

Two real deployment blockers surfaced while writing the hosting guide, and both were fixed in code rather than papered over with a documentation warning. Render hands out connection strings using the legacy postgres:// scheme that SQLAlchemy dropped, which crashes on first boot with NoSuchModuleError, so the app now normalises it. And CORS_ORIGINS only accepted a JSON array, meaning the comma separated list anyone would naturally type into a dashboard took the whole service down at startup. Both now have tests.

The adaptive engine

Every user has a rating per error type, starting at 1200 and clamped between 800 and 2000, updated with an Elo style formula after each attempt. Challenge selection weights inversely to that rating, so the error types that beat you surface more often. This is Sonkar's conditioning idea inverted: instead of generating distractors from your misconceptions, it selects challenges from them.

The duel state machine

DUEL

The ready check exists because the first version started the clock when the first player loaded the page, which handed them a head start. Now the server starts the timer only when both players have marked ready, and both receive the same started_at and expires_at. A background job settles duels whose deadline passes, and also duels with no timer whose opponent simply walked away.

Graceful degradation in the analysis report

The report runs in three separable stages: compute statistics, generate a narrative, render the PDF. If the LLM is unavailable, stage two falls back to a rule based narrative and the report still renders, because the numbers are the valuable part.

This is not theoretical. During testing I exhausted Groq's daily token limit, and the fallback carried the entire feature. It produced output like "Strongest subject is science at 100 percent accuracy" and "Lowest scoring skill is proposing the fix". Specific and useful, with no model involved.

Challenges I ran into

This is the section I would read first as a judge, because it is where you find out whether someone actually built the thing or just assembled it.

The feature that had never worked, and nobody could tell

The adaptive mastery engine was the headline differentiator. The code looked correct. It was not.

services.py selected an error type into a variable called error_type_id, but the return statement used a different variable, error_type, which was still None from its initialisation. Every challenge was saved with a null error type. Every downstream check read if challenge.error_type: and silently skipped.

I only found it because I stopped reading the code and queried the database:

challenges total = 64   has_error = True: 51   error_type NOT NULL: 0
mastery_state rows: 0   attempts: 57

Zero of sixty four. Zero mastery rows after fifty seven attempts. The entire adaptive engine had never recorded a single data point, and nothing in the UI would ever have revealed it. That one line is the difference between a demo and a product.

Duel timers were six hours wrong, and the cause was PostgreSQL

Timers showed absurd values. The obvious suspects, clock skew and serialisation, were both innocent.

The real cause: the DateTime columns are timestamp without time zone, but the duel code wrote timezone aware datetimes. When PostgreSQL receives an aware value for a naive column, it converts to the session timezone before discarding the offset. The session timezone was Asia/Dhaka, so every duel timestamp was silently stored six hours off, while the rest of the codebase used naive UTC.

The fix was a single helper, _utcnow(), returning naive UTC for every write, and _iso_utc() attaching an explicit offset on the way out so browsers stop parsing deadlines as local time. There is now a test asserting both.

Attempt lists that stopped growing after question one

Multi question duels could never complete. Players answered, and their progress stayed at one.

attempts = duel.host_attempts or []   # returns the SAME list when non-empty
attempts.append(attempt_id)
duel.host_attempts = attempts         # assigning the same object: no change detected

A plain JSON column has no mutation tracking. Mutating the list in place and assigning the identical object back leaves SQLAlchemy seeing nothing to flush. It only bit once the list was non-empty, which is why the first append worked and every subsequent one vanished. My first end to end test used a single question duel and passed, which is exactly how the bug survived.

A deadlock hiding behind a second deadlock

A player who finished first waited on the results screen forever. Two independent causes stacked:

  1. The polling interval captured the first render's closure, where the userId state was still null, so an early return fired on every tick.
  2. Even with that fixed, once the opponent completed the duel, the completion check returned should_complete: false, reason: "not_active". From the client's perspective that was indistinguishable from "still running". The terminal state looked identical to the in progress state.

The fix was to stop asking "should this complete" and start reading the duel's own status. There is now a regression test that reproduces the exact sequence.

An animation that made content invisible

The landing page reveals sections on scroll. My first implementation used IntersectionObserver, and testing caught it dropping callbacks and leaving entire sections stranded at opacity zero. Research citations and the architecture diagram were simply not there.

I replaced it with a position check on scroll plus a self cancelling poll as a backstop. Less elegant, and correct. An invisible section is a far worse failure than a missing animation, and I would rather ship the boring mechanism that always works.

Fixing a security hole and introducing a smaller one

While adding authorization I made forfeit_duel idempotent, returning the existing state if a duel was already settled. I put that early return before the participant check, so any authenticated user could call forfeit on a stranger's finished duel and receive its full details back.

My own test suite caught it, because I had written the cross user authorization tests before the implementation rather than after. Moving three lines fixed it. The lesson stuck: an idempotency shortcut is still a code path, and authorization belongs at the top of every one of them.

Accomplishments that I am proud of

A test suite that runs without spending a cent. 100 tests covering auth primitives and authorization boundaries, the points formula, taxonomy fallbacks, the duel lifecycle including UTC handling and idempotent completion, and the analysis report. The duel tests originally required live model calls, were marked as such, and were therefore skipped constantly under rate limiting, which made them useless as a safety net. I added fixtures that seed challenges directly and stub the grading call. The suite went from 82 tests that often skipped to 100 that always run, in about thirteen seconds.

A security model that assumes the client is hostile. Identity comes from a signed token on every request, never from a parameter. Attempt details, mastery data and duel results are restricted to their owner or to participants. Public profiles expose aggregate statistics only, never written answers or email addresses. Login returns byte identical responses for an unknown username and a wrong password, so it cannot be used to enumerate accounts. Legacy password hashes upgrade to bcrypt transparently on next login, so hardening cost existing users nothing.

Honest failure modes. When the grader cannot produce usable output after three attempts with escalating strictness, the API returns 503 and saves nothing, so the student can resubmit the same answer. It does not invent a score. The analysis report degrades to a rule based narrative instead of erroring. Both were designed before they were needed, and both were used.

A landing page that argues rather than decorates. It walks seven numbered chapters from the problem through the premise, the marking, the safeguards, the architecture and the evidence. Every research claim links to its source, every quotation is verbatim from the abstract, and a disclaimer separates the authors' words from my commentary. I checked all four papers against the arXiv listings and found two citation errors in my own earlier notes, which I corrected.

Measured, not assumed. Nine routes verified at four breakpoints for horizontal overflow and touch target size. Every element checked for stranded opacity. The nav alignment fixed by measuring pixel offsets rather than eyeballing.

What I learned

Reading code is not verifying code. The mastery engine looked right in review and was dead in production. One database query answered in seconds what an hour of reading had not. I now reach for the data first.

Documentation drifts into fiction. The project I inherited from my own earlier sessions had documents declaring "production ready", "zero TypeScript errors" and "mobile complete", while the frontend did not compile and the entire duel feature returned 500 on every read. Twenty one markdown files, each written at a moment of optimism, none checked afterwards. I deleted them and now treat any status claim as a hypothesis until a command proves it.

A passing test can be a lie about coverage. My single question duel test passed while multi question duels were fundamentally broken. Tests that skip under rate limiting provide no safety at all while looking green.

Correct beats elegant when the failure mode is invisible. IntersectionObserver is the right tool on paper. It also silently hid my content. The boring scroll handler ships.

Fixing security requires the same paranoia as writing it. My idempotency shortcut opened a hole inside the very commit that closed a bigger one.

What is next for Gotcha!

Near term, and honest about it. There is no migration framework, so schema changes on a deployed database are manual. Alembic is the first thing on the list. There is no rate limiting, and challenge generation is unauthenticated, which is a real quota risk. The streak counts consecutive correct answers rather than consecutive days, which does not match how it is described.

Product. A teacher dashboard showing which error types a class collectively struggles with, since the per error type data already exists and is exactly what an instructor would want. A second grader model to cross check marks on borderline answers. Custom taxonomies so a department can define the mistakes that matter in their subject.

Research. The mechanism from Sonkar and colleagues suggests a real evaluation: generate distractors conditioned on a specific student's recorded misconceptions and measure whether they select them at rates comparable to expert written ones. The data model already captures what that study would need.

Why this deserves first place

I would rather argue this from evidence than from adjectives.

It answers a question the field is actively asking. Four papers from 2025 and 2026 independently propose teaching with deliberately flawed AI output. None of them ship a system. This is that system, and each paper maps onto a specific mechanism: Wazan to controlled uncertainty in generation, Hosseini to error as curriculum, Lamberti to the locate, diagnose and fix loop, Sonkar to the adaptive engine.

The core insight is architectural, not cosmetic. Separating generation from grading is what makes the score trustworthy. Sealing ground truth server side is what makes the game honest. Splitting the mark into three axes is what makes feedback actionable. These are design decisions with consequences, not features on a list.

It is engineered, not assembled. Row level locking so concurrent duel completions cannot double pay a wager. A background job that settles abandoned duels, including untimed ones. Transparent password hash upgrades. Escalating retry strategies with a documented failure mode. Graceful degradation that was exercised for real when a rate limit hit mid demo.

The bugs I found are the proof. A dead adaptive engine that no interface would reveal. A six hour timezone shift caused by PostgreSQL session conversion. Silent JSON column mutation loss that only appeared past question one. None of these are surface bugs. Finding them required querying production data, reading driver behaviour and testing the case that had not been covered. That is the difference between a project that demos and a project that works.

It states its own limitations. The README has a "Known limitations" section listing the missing migration framework, the absent rate limiting, the streak semantics mismatch and the single grader dependency. I would rather a judge read that from me than discover it themselves.

Most hackathon submissions are a good idea with a demo path. This one has a research thesis, an architecture that follows from it, a test suite that protects it, and a written account of everything still wrong with it.

Try it: pick a subject, read carefully, and see whether the mistake gets past you. It usually does the first time.

Built by ahammadshawki8

Built With

Share this project:

Updates