Show Your Marks

A PLAB 2 history-taking station with a simulated patient, where every mark in your report opens the question that earned it, and the mark scheme is public.(access code: plab2026)

Comment

Inspiration

International doctors who want to work in the UK sit PLAB 2: sixteen 8-minute stations with actors playing patients. The exam fee is £1,036, and practice usually means a partner on a video call or an academy. In the practice tools I could find, there's either no feedback or the marking is a black box: you get a score and can't see why. Several dozen AI patient products already exist, so another chatbot that plays a patient wasn't the gap. A patient that stays consistent, and feedback you can check, was.

What candidates told us

Before building further I read 123 public Reddit threads where PLAB 2 candidates discuss how they prepare (January 2025 to October 2026), with an AI research assistant, and mapped each pain point to what the design does about it. The full write-up, with a link to every thread, is in the repo at docs/user-research.md.

  • Partners cancel, and practice depends on finding someone. The tool runs alone, any time, on a phone.
  • AI patients contradict themselves: one candidate's AI patient changed its presenting complaint mid-station. Here the patient only holds the facts you've unlocked, and they're fixed in the case file.
  • Real role players wait for "the magic question", in one candidate's words. That's the core of this design: facts unlock only when you ask.
  • Marked down for something they did ask. Every mark here opens the line that earned it, and the rules are public.
  • Cost. Free, no signup.
  • Recalled exam questions circulate widely, and in September 2026 two academy tutors were suspended over sharing exam content. Every case here is built from published, openly licensed mock consultations.

What it doesn't answer, and the research is clear these matter: management and timing are the most common reasons people describe failing, candidates need to practise out loud, and sounding scripted costs rapport marks. This tool covers history taking only, it's typed, and it can't judge tone. The trusted resources candidates recommend to each other are free community notes, a lecture course, a prescribing course and human partners; AI tools come up as the stand-in when no partner is available, and that's the honest place for this one too.

What it does

You pick a station (chest pain, back pain, nausea, diarrhoea), read the brief, and take a history by typing. On a phone you can use your keyboard's microphone instead. The patient only tells you what you actually ask about, and keeps one worry to themselves until you ask about their concerns or build some rapport.

When you finish, or the 8 minutes run out, the report leads with the result: how much you covered, which critical item you missed, and whether the hidden concern came out. Every covered item quotes the question that earned it. Click it and the full conversation opens with that line highlighted. Every missed item has an example of how to ask it and shows where the criterion comes from: GMC PLAB 2 guidance, Calgary-Cambridge, or "this case's own" when nothing published backs it. The GMC's ten feedback statements are there too, each one saying "needs attention", "no concern" or "not assessed here", because a typed station can't judge your tone of voice and shouldn't pretend to.

No code? The button at the top of the page shows a finished report from a real case in one click.

How we built it

Three Nemotron 3 Super calls per question, all on Nebius Token Factory:

  1. The judge reads your latest line against the case's published rubric and returns structured JSON: which rubric items it covers, question style, empathy, interruptions, jargon, and whether the line was readable at all. It never sees the case facts.
  2. The patient gets a prompt built from the facts you have unlocked so far, and nothing else. Facts you haven't asked about are not in its context at all, so no jailbreak can pull them out. An output guard then checks the reply for still-locked terms, retries once, and redacts if needed.
  3. The safety check runs beside the judge and pauses practice if you seem to be describing your own real symptoms, pointing you to NHS 111, a GP, 999 or Samaritans.

Which facts unlock is decided by code, not by a model: pure functions with unit tests, 987 of them across the repo. The judge's output is validated with Zod before it touches state, and anything it returns that isn't a real rubric id is dropped.

I compared five judge settings on the same 50 labelled lines before choosing one. Nemotron 3 Nano recalled 17% of rubric items. Nemotron 3.5 Lightning was fast but credited items on 40% of the off-topic and injection lines, which would unlock facts nobody asked for. Super with reasoning off had 0% false unlocks at about a second a line. Ultra and Super with reasoning on scored higher (96–98% exact lines) but cost 2.5–3.5 times as much and Super-with-reasoning took over three seconds a line, which is too slow for a conversation. So Super with reasoning off runs everything. A full station costs well under a cent.

The cases are built from published, openly licensed mock consultations (Fareez et al. 2022, CC0; PriMock57, CC BY 4.0). A pipeline proposes facts with a model, and code keeps a fact only if its exact sentence is found in the transcript with a line number, so nothing is invented.

The site is a plain Node server with the whole station state sealed in encrypted cookies (AES-256-GCM), so nothing is stored on the server. It's on Render's free tier, with spend caps in code: $0.05 per station, $0.50 per day, 40 questions per station, and an access code. Token Factory has no hard spending limit of its own, so these caps are the safeguard. /healthz reports today's spend against the cap.

Measured, and published

All on one synthetic case, one run each, so read them as direction, not rates. The tables, the test sets and every line are in eval/results/.

  • Leaks under attack: 24 jailbreak and trick lines. A control version with every fact in the prompt and an instruction not to volunteer them leaked 1.33 locked facts per attack (58% of attacks leaked something). This engine leaked 0.04 per attack before the output guard and 0 as shown to the candidate.
  • Invented answers: asked about things the case never mentions, the patient model answered with unsupported content 70% of the time. Three prompt rewrites moved that to 55% and back to 65%. What fixed it was code, not prompting: the judge flags lines that ask for case information, and the engine re-asks and then replaces an answer the case can't support. Measured after: 0% of 20, with 13 replies caught and re-asked on the way.
  • Safety pause: on a held-out set of 57 self-disclosures and 152 ordinary candidate lines, the rule list alone caught 40%. Rules plus the Super safety check caught 100% with 2% false pauses on candidate lines. That set was used to revise the prompt once, so treat the recall as an upper bound.
  • Judge accuracy: 94% of lines exactly right, 0% false unlocks, in the latest run of the shipped setting. Two runs of the identical measurement differed by two points, so that's the noise.

Challenges we ran into

  • The prompt couldn't stop the patient making things up. Three careful rewrites of the honesty instructions moved the invention rate by two or three answers out of twenty, and one made it worse: the model started decorating denials ("No chest pain. Just this stubborn cough."), which a candidate would write down as history. The fix was to move the rule out of the prompt and into the engine.
  • A live bug found during testing. Typing "?" or a stray fragment made the patient model carry the conversation on by itself: it asked the doctor's questions and invented symptoms the case never gave it. Root cause: a line with no question reached the patient model with nothing to constrain it. Now punctuation-only lines are refused before any model call, the judge marks unreadable lines, and those get a fixed "Sorry, I didn't catch that" without the patient model being called.
  • Plans changed under me. I started with a multimodal idea, but Nemotron 3 Nano Omni was taken off Token Factory's serverless tier on 31 August, and Token Factory has no speech model, so this became a text product.

Accomplishments that we're proud of

  • A patient that can't leak what it was never given, with the attack numbers to show it.
  • Marking you can argue with: the rubric, its sources and the unlock rules are public code, and every mark in the report links to your own words.
  • Saying what it can't do. The report marks what a typed station can't judge as "not assessed here" instead of guessing.

What we learned

Measure before you trust a prompt. Several things I was sure the prompt handled turned out not to, and the fixes that held were in code, where a test can check them. Also, the cheapest model wasn't the cheapest choice: a judge that unlocks facts on off-topic lines would quietly ruin every station.

Accessibility

The station works on a phone, and you can answer by voice through your keyboard's dictation, so you can practise speaking, not only typing. All text is real text, not images. I have not tested it with a screen reader yet.

Known limitations

  • Typed, not spoken. PLAB 2 is a spoken exam, and typing doesn't practise tone, pace or non-verbal cues; the report says so.
  • No case has been reviewed by a clinician. Every case and every report says so, and there's a link to report clinical errors.
  • 4 real cases plus one synthetic demo. History taking and communication only; no examination, diagnosis or management marking.
  • The evaluation is one synthetic case, one run per line. No inter-rater agreement figure against a human examiner yet.
  • The judge sometimes treats rude or one-word lines ("stupid", "hungry") as unreadable instead of reacting to them.
  • A question that asks about something in the case and something outside it in one breath ("Can you describe the pain? Does anything make it better?") can draw an invented denial for the second half. The honesty guard works per line, not per part of a line.
  • Educational use only. It is not a medical device and gives no medical advice.

What's next

Find a clinician to review the cases, measure agreement between the judge and human examiners on real transcripts, and add cases. And speech, once there's a speech model on the same platform: NVIDIA's open speech recognition models would fit right in.

Built With

Share this project:

Updates

Submission history