Inspiration
My inspiration for EvalGuard comes directly from my experience as a Graduate Teaching Assistant in Artificial Intelligence, where I instruct 150+ students on data preprocessing, feature engineering, and model validation. In practical labs, I often see students write machine learning code that runs successfully but produces results that are not actually reliable.
The most dangerous mistakes are not always syntax errors. They are silent ML evaluation bugs: data leakage, using accuracy on imbalanced datasets, or accidentally using test data during model selection.
For example, in an imbalanced A/B testing problem, if only 5% of users convert, a model can predict “no conversion” for every user and still get:
$$ \text{Accuracy} = 95\% $$
That looks good, but it completely fails the real goal.
My background in projects like early Alzheimer’s detection, explainable AI, and anomaly detection in financial services also shaped this idea. In those projects, I worked with class imbalance, recall, ROC AUC, cross-validation, and model trust. EvalGuard was built from that same lesson: machine learning is not just about getting a high score, it is about knowing whether the score is valid.
What it does
EvalGuard is a Socratic ML debugging lab partner for students and teachers.
A user can upload or paste a Python machine learning script, optionally upload a dataset, and let the app analyze the workflow. EvalGuard detects silent ML bugs such as:
- data leakage before train/test split,
- accuracy misuse on imbalanced datasets,
- validation leakage during hyperparameter tuning or model selection.
After analysis, EvalGuard shows:
- the detected bug type,
- confidence score,
- evidence lines from the code,
- a Socratic hint,
- why the issue matters,
- what the student should inspect next,
- a generated validation test,
- sandbox test execution output,
- a downloadable teacher report.
The key idea is that EvalGuard does not simply give away the fixed code. It helps students reason toward the fix.
How I built it
I built EvalGuard using Python and Streamlit.
The app has a simple workflow:
- The student or teacher uploads a script.
- The diagnosis engine checks the code for silent ML bugs.
- If enabled, the app can use an OpenAI model for structured diagnosis.
- If the API path is unavailable, it falls back to local rule-based detection.
- The app generates a pytest-style validation test.
- The generated test runs inside a sandbox.
- Teacher mode creates an exportable report with evidence, explanation, and a student-facing question.
I also created demo cases for:
- leaky preprocessing,
- A/B testing metric misuse,
- validation leakage.
I used Codex throughout the development process to help build the Streamlit workflow, improve the product structure, generate sample test files and datasets, add .env support, implement Teacher mode, and verify the project with automated tests.
Challenges I faced
One major challenge was balancing AI assistance with academic integrity. A typical AI assistant might directly rewrite the student’s code, but that can reduce learning. Since my goal was to support education, I designed EvalGuard to give Socratic hints instead of complete answers.
Another challenge was making the feedback evidence-based. I did not want the app to only say, “This might be wrong.” I wanted it to show why, generate a test, run the test, and provide a teacher-readable explanation.
A third challenge was designing for both students and instructors. Students need guidance that helps them think. Teachers need clear evidence, reports, and a way to explain why a mistake matters.
What I learned
I learned that a strong educational AI tool should not always optimize for the fastest answer. Sometimes the better tool is one that asks the right question.
Through building EvalGuard, I also reinforced an important machine learning principle: a model score is only meaningful if the evaluation process is valid.
My teaching experience, AI research projects, and full-stack development background all came together in this project. EvalGuard combines ML validation, responsible AI tutoring, test generation, and a usable web interface into one learning-focused tool.
What is next
Next, I want to improve EvalGuard with:
- AST-based code analysis instead of simple pattern detection,
- notebook upload support,
- line-level code highlighting,
- more ML bug classes,
- richer dataset-aware generated tests,
- instructor dashboards for multiple submissions,
- better classroom grading and feedback workflows,
- stronger support for explainable AI and imbalanced learning cases.
Built With
- ai
- codex
- data
- edtech
- education
- gpt-5.6
- machine
- openai
- pytest
- python
- science
- scikit-learn
- streamlit
Log in or sign up for Devpost to join the conversation.