ScholarPanel

ScholarPanel questions students and researchers on their own work, the way a real panel would, so they rehearse a thesis defense or an academic interview before facing the real one.

Comment

ScholarPanel

Inspiration

I was my supervisor's first PhD student. He had come from another institution, had never taken a student through a defense, and did not work closely with the internal committee members who might have shown me how one goes. My preparation for my first doctoral exam, July 2023, was one mock talk in my lab and whatever I read on my own. Then I sat across from four examiners in four specialties, each coming at me from an angle I had not considered.

By my proposal defense in 2025 the models had gotten good, so I uploaded my thesis and ran deep research on several of them, asking each one what my committee would ask. The questions helped. They also failed the same way twice. They stayed close to experimental detail while my committee went broader and stranger than anything a model produced, and they arrived as a list that never changed. A real examiner hears your answer and asks the next question because of what you just said. What I had instead was a loop of copying a question out of a chatbot, recording myself, pasting the answer back, and hoping it followed up before it lost the thread. At one defense somebody asked whether, knowing what I know now, I would have run the whole PhD differently. Nothing on any generated list had ever put that in front of me.

Then the faculty job market did it again. My first screening interview lasted ten minutes, at a university I badly wanted. I had spent years being trained to answer thoroughly, so I answered thoroughly, and the panel ran out of time before it finished its questions. I knew my work cold. I had rehearsed the subject and never the format.

I assumed this was my problem. It is not. Every graduate student and postdoc I asked told me a version of the same story, and even the ones with generous supervisors got exactly one rehearsal, because nobody has the hours to sit and drill follow-up after follow-up.

There is a technical reason it stays unsolved. My final thesis ran past 300 pages. Hand that to a frontier model and ask for questions, and it cannot tell your methodology from your contribution, it drifts long before the hour is up, and it certainly cannot be four examiners in four fields at once.

What it does

You bring your own work. A thesis chapter, a research statement, a paper. ScholarPanel reads it, finds the claims and the places a committee would push, and asks you about them out loud. You tell it what you are preparing for, whether that is a thesis defense, a faculty job talk, or an admissions or fellowship interview, and you pick one to three panel roles. When you answer, it asks the follow-up your answer invites. Afterward you get feedback tied to what you actually said, linked back to turns in your transcript, with something specific to fix.

That follow-up is the whole thing. A fixed list of questions cannot hear you overreach on a claim your evidence does not support and press you right there. ScholarPanel can, because the questions come out of passages you wrote, and it is reacting to the answer sitting in front of it.

How we built it

The idea and product direction were mine. The build was the two of us, me and the other team-member.

Python and FastAPI on the backend, Next.js and TypeScript on the front, Postgres, and a pgvector store with Voyage embeddings for retrieval over whatever material the user has selected. Every model call goes through one typed OpenAI Responses API client. GPT-5.6 Terra handles the live questioning, document analysis, and extraction. GPT-5.6 Sol handles scoring and the final write-up. Anything schema-bound uses Structured Outputs. The client also carries bounded retries, output and spending limits, per-user database isolation, and usage records that keep no prompts or responses.

For the Build Week extension we worked inside Codex with GPT-5.6 doing the reasoning, and we were strict about who decided what. The product calls stayed with us, everything that decides what ScholarPanel is, from the preparation-only boundary and the Education framing to the five preparation paths, the one to three panel roles, the wording for each situation, the provider choices, and how much to ship. Codex took the work that rewards patient reading of a repository, and it traced the existing model call sites, stood up the typed Responses API runtime, moved the schema-bound calls onto Structured Outputs, wrote the failure-path and usage tests, ran the migrations and the full suite, drove the app in a browser, walked the error paths, and assembled the evidence record. You can check any of that against dated commits, pull requests, test files, and CI runs.

One more line we drew. ScholarPanel existed before the submission period opened, so the evidence record separates the baseline commit from the Build Week work and claims only the latter. A full-history secret scan ran before freeze.

Challenges we ran into

Retrieval had to stay honest, and at first it did not. The panel would ask a question that sounded excellent and had no support anywhere in the candidate's document. Binding each question to retrieved passages from the selected material fixed it, along with keeping claim extraction separate from the questioning, so the pressure lands where the evidence is actually thin.

Holding a role was harder than either of us expected. One assistant collapses into one generic voice. Getting a chair, a supervisor, and an external examiner to sound different and care about different things, all out of the same engine, meant keeping the internal session model neutral and moving the situation-specific language out into configuration the frontend reads at render time.

Proving it worked was its own project. For a submission judged on real software, "it works" is worth nothing, so we paid for the checks. A Terra Structured Outputs extraction on our own thesis and papers. A Voyage embedding call. Hosted Terra question generation grounded in that material. A hosted Sol scoring run that came back with five category scores, a written summary, methodological gaps tied to specific transcript turn IDs, and five improvements. Where we could not prove something on the judge deployment, like the microphone path, we wrote that down too.

Accomplishments we're proud of

The follow-up works. Every chatbot workaround I tried during my own defenses broke at exactly the same spot, where a weak answer should have been challenged and instead the thing moved politely on to the next topic. In a ScholarPanel session, the examiner role catches an overreached claim and asks where the evidence stops, on a document it read minutes earlier. That is the closest anything has come to the room I actually walked into.

The product runs all the way through. Sign in, add your work, pick a situation and up to three roles, practice through the follow-ups, end the session, read feedback tied to your own transcript. We ran that full sequence on the hosted deployment, through paid scored feedback, using real academic material. On a fresh database built from every migration, 463 backend tests and 239 frontend tests pass.

I am equally proud of what we did not claim. We paid for real provider checks rather than asserting the calls worked. We scanned the entire commit history for secrets. We stated plainly where we have no proof, including any measured learning outcome. All of it is checkable.

What we learned

Most of the hard part had nothing to do with the model. Getting a language model to write a plausible exam question is trivial in 2026. Keeping the questions inside the candidate's own evidence is not. Neither is switching voice cleanly from supervisor to external examiner, or following up on one weak answer without wandering off, or tying a piece of feedback back to a real turn in the transcript. Claim extraction, retrieval, the role definitions, the evidence-linked scoring, each of those had to be built and tested by itself before a session started to feel like a panel.

The other surprise was how much discipline the safety boundary demanded. This product sits one design decision away from being a live-exam-help tool, which is the one thing it must never become. Every screen had to stay on the practice side of that line, and we wrote the rule into the repository so no shortcut could quietly cross it later.

What's next

An education-outcomes study. Right now we claim a working preparation method and a design that pushes candidates on their own evidence. We do not claim measured efficacy. Does a candidate who rehearses this way state their contribution more clearly? Do they hold the thread through a hostile follow-up? I do not know yet, and consented pilots across the five preparation paths are how we would find out.

I built the tool I wanted at 2am the week before my proposal defense, sitting with a 300-page document and no one to ask me about it.

Built With

Share this project:

Updates