Scivet: Making Student Thinking Visible
Inspiration
Traditional science assessments usually focus on whether a student reaches the correct answer. However, a correct answer does not always demonstrate real understanding, and an incomplete answer does not necessarily mean that the student lacks scientific reasoning.
One-on-one interviews can reveal how students interpret evidence, connect scientific concepts, revise misconceptions, and transfer knowledge to new situations. Unfortunately, these interviews are difficult to scale, time-consuming for teachers, and challenging to evaluate consistently.
I built Scivet to explore a different approach: an AI-powered science assessment platform that treats assessment as an adaptive conversation rather than a static test.
Scivet is grounded in Evidence-Centered Design (ECD), an assessment framework that connects what we want to measure, what observable evidence would demonstrate it, and what tasks can elicit that evidence:
$$ \text{Student Response} \rightarrow \text{Observable Evidence} \rightarrow \text{Competency Inference} $$
The goal is not to replace teachers. It is to give them a scalable way to uncover student thinking while keeping the assessment process transparent, evidence-based, and open to human review.
What Scivet does
Scivet supports the complete lifecycle of a conversational science assessment—from designing the task to interviewing students and generating an explainable report.
1. Multi-agent assessment design
A teacher begins with a topic, learning objective, source material, or a simple description of an assessment idea. Scivet then coordinates several specialized AI agents:
- The Domain Agent identifies core concepts, causal mechanisms, relevant variables, partial understandings, and common misconceptions.
- The ECD Agent converts the domain analysis into a student model, evidence model, task model, and scoring rubric.
- The Task Agent generates an age-appropriate scientific scenario, supporting materials, an open-ended initial question, and follow-up strategies.
- The Critic Agent reviews the generated blueprint and task for weak evidence alignment, inappropriate difficulty, validity risks, and other quality issues.
- Teachers can inspect, edit, approve, and publish the result before students see it.
This creates a human-in-the-loop workflow instead of asking one model to generate an entire assessment in a single opaque response.
2. Adaptive AI interviews
Students respond through a conversational interface. The Dialogue Agent analyzes each response and decides what to ask next.
Rather than following a fixed questionnaire, it can:
- Ask students to clarify their explanations.
- Request evidence from the provided material or data.
- Probe the relationship between evidence and conclusions.
- Present a plausible misconception and ask the student to evaluate or correct it.
- Introduce a new condition to test whether the student can transfer their reasoning.
The final follow-up moves the explanation into a related situation, allowing Scivet to examine whether the student understands the underlying mechanism instead of memorizing a specific answer.
3. Evidence-grounded assessment
After each response, the Evidence Agent extracts observable evidence from the student's own words. It identifies:
- Claims
- Scientific concepts
- Evidence
- Reasoning
- Misconceptions
- Transfer of understanding
Every extracted signal is connected to a verbatim quotation from the student. Scivet does not only produce a score; it shows the evidence behind that score.
The Scoring Agent evaluates the evidence against an ECD-aligned rubric. Each dimension receives a score, supporting quotations, a rationale, and a confidence indicator.
Finally, the Critic Agent reviews the scoring result for insufficient evidence, overconfident conclusions, or validity risks. Uncertain cases can be flagged for human review.
4. Explainable reports
Students and teachers receive a structured report containing:
- Dimension-level performance
- Evidence annotations from the conversation
- Strengths and areas for growth
- Detected misconceptions
- Personalized next steps
- A transparent explanation of how the evidence supports the evaluation
Teachers can also perform a manual review, compare evidence across submissions, and export assessment results.
How I built it
Scivet uses a modular multi-agent architecture. Each agent has a clearly defined responsibility, input schema, and output schema. The agents communicate through structured JSON rather than unstructured prose, making the pipeline easier to inspect, validate, and improve.
The AI layer uses an OpenAI-compatible Chat Completions interface. Prompts are separated by agent role and can be versioned and managed through the administration interface. Structured outputs are validated before they enter the next stage of the pipeline.
The application is built with:
- Next.js and React for the full-stack web application
- TypeScript for type-safe agent contracts and application logic
- OpenAI API for task generation, adaptive dialogue, evidence extraction, scoring, and critique
- Zod for validating structured agent outputs
- Prisma and SQLite for assessments, conversations, reports, prompts, and user data
- Redis and BullMQ for asynchronous AI tasks and production-scale queues
- NextAuth for role-based authentication
- Tailwind CSS for the interface
- Recharts for assessment visualizations
- PDFKit for downloadable reports
The platform also records agent runs, warnings, confidence values, and intermediate outputs so that important assessment decisions remain auditable.
Challenges I faced
Keeping the interview adaptive without making it leading
An AI interviewer can easily reveal the expected answer through its questions. I had to design dialogue policies that respond to missing evidence while avoiding phrases that expose the rubric or tell students what conclusion to reach.
Connecting scores to real evidence
Free-form AI feedback can sound convincing even when it is not grounded in what the student actually said. To address this, Scivet requires evidence extraction and scoring outputs to include verbatim student quotations. Missing evidence results in lower confidence or a human-review flag.
Maintaining reliable communication between agents
Multi-agent systems become fragile when one agent produces an unexpected structure. I introduced explicit schemas, runtime validation, bounded critic-and-revision loops, JSON repair handling, and clear information boundaries between agents.
Balancing automation with assessment validity
Educational assessment is a high-impact use case. A fully automated score should not be treated as unquestionable truth. Scivet therefore exposes evidence, records limitations, supports teacher editing, and routes uncertain cases to human review.
Building for both local demos and production
The application needed to remain easy to run during development while supporting longer AI workloads in production. Scivet can run synchronously with a local SQLite database, or use Redis and BullMQ workers for queued model calls and concurrent users.
What I learned
The most important lesson was that adding more agents is not automatically useful. Multi-agent collaboration becomes valuable when each agent has a distinct responsibility, a constrained output, and a clear handoff to the next stage.
I also learned that explainability in education must be more than a generated paragraph. A meaningful explanation needs traceable evidence, explicit criteria, uncertainty indicators, and a path for human correction.
Finally, prompt design and software architecture cannot be separated in an AI-native product. Schemas, validation, retry policies, interface design, and human review are all part of the model interaction—not just infrastructure around it.
What I am proud of
Scivet transforms assessment from a final score into a visible reasoning process. It can show not only what a student answered, but also:
- Which evidence they used
- How they connected evidence to a claim
- Which misconception they recognized or retained
- Whether they could revise their explanation
- Whether their understanding transferred to a new situation
The result is a more informative experience for students and a more scalable diagnostic tool for teachers.
What's next
The next steps are to:
- Pilot Scivet with science teachers and real classroom tasks
- Compare AI-generated scores with independent human reviewers
- Add calibration and inter-rater reliability analytics
- Support multimodal assessments involving diagrams and experiment images
- Add voice-based science interviews
- Expand to additional languages, grade levels, and scientific domains
- Build teacher-facing analytics for identifying class-wide misconceptions
Scivet's long-term vision is to make high-quality formative assessment feel less like grading a test and more like listening carefully to how every student thinks.
Log in or sign up for Devpost to join the conversation.