WISE — Workflow for Interpretable Scientific Evaluation

Smarter model selection for reliable, explainable science.

Comment

WISE-XAI — Workflow for Interpretable Scientific Evaluation

Inspiration

Machine-learning, deep-learning, and advanced AI models are becoming increasingly powerful and accessible. However, responsible model selection is still difficult.

Students and scientists often choose a model because it is new, popular, or reported to achieve high accuracy—not because it fits their dataset, research objective, validation structure, or need for interpretation.

A complex model may produce an impressive score while overlooking temporal leakage, repeated observations, limited sample size, missing data, class imbalance, or unstable feature relationships. More importantly, the model with the highest accuracy is not always the best scientific model. Depending on the research objective, interpretability, reliability, reproducibility, and applicability may be more important than a small improvement in predictive accuracy.

We built WISE-XAI around a simple principle:

Start with the data and the scientific question—not with the newest model.

WISE-XAI is an educational guidance platform that helps students and scientists diagnose their data, compare appropriate models, design valid evaluation strategies, interpret results, and generate reproducible scientific outputs.

What WISE-XAI Does

Users can upload a CSV dataset or use a built-in scientific demonstration dataset. They then define:

  • Their research question
  • The target variable
  • Regression or classification
  • An optional temporal variable
  • An optional grouping variable
  • Optional latitude and longitude variables

WISE-XAI organizes the analysis into seven connected stages.

1. Data Overview

WISE-XAI examines sample size, feature count, missing values, duplicate records, possible identifier variables, the sample-to-feature ratio, and available temporal or grouped structure.

It then provides initial guidance on model complexity. A more advanced model is not recommended simply because it is available; its capacity should be supported by the amount, quality, modality, and structure of the training data.

2. Target Insights

WISE-XAI visualizes the target distribution, temporal behavior, and spatial pattern before model training.

These visualizations help users identify skew, class imbalance, trends, clustering, and other structures that may affect validation and interpretation.

3. Feature Relationships

WISE-XAI displays a lower-triangle correlation heatmap, feature–target association scores, and a ranked feature-association table.

For regression, predictors are initially screened using absolute correlation with the target. For classification, mutual information is used.

Correlation and feature association are useful for exploration, but they do not establish causality.

4. Validation Design

The current prototype supports:

  • Random holdout
  • Group-based holdout
  • Temporal holdout

For temporal data, WISE-XAI trains on the earliest 80 percent of observations and tests on the latest 20 percent. This helps prevent future-information leakage.

For grouped data, WISE-XAI separates groups between training and testing so that related observations do not appear in both sets.

The broader framework is designed to support advanced evaluation methods such as stratified, repeated, grouped, nested, time-series, spatial-block, and spatiotemporal cross-validation.

5. Model Comparison

WISE-XAI is designed as a model-agnostic framework for comparing statistical models, conventional machine-learning methods, deep-learning architectures, and emerging AI models.

The current prototype demonstrates the complete decision workflow using:

  • Multiple Linear Regression or Logistic Regression as a transparent baseline
  • Random Forest as a nonlinear comparison model

These models were intentionally selected to make the prototype clear, reproducible, and easy to evaluate.

The broader framework can support:

  • Linear and generalized linear models
  • Decision trees and ensemble methods
  • Gradient-boosting models
  • Explainable Boosting Machines
  • Support Vector Machines
  • Neural networks
  • CNNs and recurrent models
  • Transformers
  • Domain-specific and emerging AI models

WISE-XAI does not assume that the newest or most complex model is automatically the best. It evaluates whether additional complexity is justified by the data structure, validation results, scientific objective, and interpretability requirements.

In the built-in demonstration, Multiple Linear Regression outperforms Random Forest on the temporal test set. WISE-XAI therefore recommends the simpler and more interpretable model.

6. Explainability

WISE-XAI currently provides:

  • Transparent baseline-model interpretation
  • Permutation importance
  • SHAP

Permutation importance measures how much held-out performance decreases when a feature is shuffled. SHAP shows how feature values contribute to individual model predictions.

The broader framework can also support partial dependence, Explainable Boosting Machine response curves, and model-specific deep-learning interpretation methods.

A variable can be important to a prediction without being proven to cause the outcome.

7. Scientific Recommendation and Outputs

WISE-XAI produces:

  • A recommended primary model
  • A model-selection rationale
  • Validation guidance
  • Selected predictors
  • Held-out performance comparisons
  • Explainability results
  • Editable starter Python code
  • A presentation-ready scientific summary figure

Prototype Scope

This submission presents an initial working prototype of the broader WISE-XAI framework.

The prototype focuses on:

  • Tabular regression and classification
  • Linear or logistic baselines
  • Random Forest comparison
  • Random, grouped, and temporal holdouts
  • Correlation and feature–target association analysis
  • Permutation importance and SHAP
  • GPT-5.6 scientific interpretation
  • Reproducible starter code
  • A presentation-ready summary figure

The prototype intentionally uses a limited number of models to demonstrate the complete model-selection workflow clearly. WISE-XAI is not conceptually limited to these models. Its architecture is intended to support broader ML and DL libraries, recent model families, domain-specific methods, and advanced cross-validation approaches.

How We Built It

WISE-XAI combines deterministic data analysis, an extensible model-comparison framework, and AI-assisted scientific guidance.

The application was developed using:

  • Python
  • Streamlit
  • pandas
  • NumPy
  • scikit-learn
  • Plotly
  • Matplotlib
  • SHAP
  • OpenAI API
  • GPT-5.6
  • Codex
  • GitHub

The prototype includes six main components:

  1. Dataset profiling
    Examines sample size, feature count, missingness, duplicates, identifiers, temporal structure, grouping structure, and possible validation risks.

  2. Feature and target analysis
    Visualizes target behavior, temporal and spatial context, correlations, and feature–target associations before model selection.

  3. Model-comparison engine
    Compares transparent baselines with Random Forest and provides a modular foundation for additional statistical, ML, DL, and emerging model families.

  4. Validation engine
    Implements random, temporal, and group-based holdouts and provides a foundation for more advanced cross-validation.

  5. Explainability layer
    Uses transparent-model interpretation, permutation importance, and SHAP while distinguishing model explanation from causal inference.

  6. Reproducible output layer
    Produces model comparisons, scientific recommendations, editable starter code, and a presentation-ready summary figure.

How We Used GPT-5.6

GPT-5.6 powers the scientific interpretation layer.

It receives only summarized diagnostics and model results—not the complete uploaded dataset—and translates the deterministic Python analysis into concise educational guidance covering:

  • Recommended model
  • Model-selection rationale
  • Validation considerations
  • Interpretation
  • Limitations
  • Recommended next steps

The numerical analysis remains deterministic in Python. GPT-5.6 makes the results easier for students and scientists to understand without replacing the underlying statistical workflow.

How We Used Codex

Codex supported development and refinement of the application.

It was used to review and improve:

  • Streamlit application structure
  • Model-comparison logic
  • Visualization code
  • Error handling
  • GPT-5.6 integration
  • Syntax validation
  • Debugging and code refinement

This combination allowed WISE-XAI to connect reproducible Python analysis with advanced AI-assisted explanation and software development.

Challenges We Faced

Avoiding Generic Recommendations

There is no universally best model. A recommendation must depend on sample size, feature count, task type, data modality, validation structure, predictive performance, interpretability needs, and scientific objective.

WISE-XAI therefore evaluates model suitability and complexity rather than automatically selecting the most advanced algorithm.

Preventing Validation Leakage

A strong performance score can be misleading when the validation design does not reflect the structure of the data.

Temporal, grouped, spatial, or repeated observations require different evaluation strategies from independent random samples. Designing WISE-XAI to recognize these structures was essential for scientifically defensible recommendations.

Balancing Automation and Transparency

We did not want WISE-XAI to become an opaque AutoML system.

The platform automates repetitive analysis while keeping the workflow visible and editable. Users can inspect the selected variables, validation strategy, model metrics, explanations, and generated starter code.

Comparing Simple and Advanced Models Fairly

Advanced models can improve predictive accuracy, but the gain may be small, unstable, or scientifically difficult to interpret.

WISE-XAI must compare accuracy together with complexity, validation reliability, applicability, and interpretability.

Separating Prediction, Explanation, and Causality

Predictive accuracy, model interpretability, and causal understanding are related but different concepts.

WISE-XAI repeatedly communicates these distinctions so that users do not treat correlation, feature importance, or SHAP values as causal evidence.

Integrating GPT-5.6 Safely

GPT-5.6 needed enough information to generate useful guidance without receiving the complete user dataset.

We therefore designed the application to send only summarized diagnostics, model metrics, selected variables, validation information, and explainability results.

What We Learned

The most important lesson is that model selection is not simply an optimization problem. It is a scientific-design decision.

A simpler model with realistic validation and understandable behavior may provide stronger scientific evidence than a highly complex model with slightly better—or even worse—performance.

We also learned that explainable AI should begin before the explanation plot. It begins with:

  1. Understanding the research objective
  2. Diagnosing the dataset
  3. Selecting an appropriate validation design
  4. Establishing a transparent baseline
  5. Comparing suitable candidate models
  6. Justifying additional model complexity
  7. Interpreting the final model carefully

SHAP is useful, but it cannot correct a poorly designed experiment or an inappropriate validation strategy.

What’s Next

Future development will extend WISE-XAI into a broader model-selection and scientific-evaluation platform.

Planned capabilities include:

  • Automated comparison across larger ML and DL model libraries
  • Integration of recent and domain-specific models
  • Stratified, grouped, repeated, nested, temporal, spatial, and spatiotemporal cross-validation
  • Hyperparameter optimization within valid training folds
  • Image, text, time-series, spatial, and multimodal data
  • CNN, recurrent, transformer, and foundation-model workflows
  • Forecasting and uncertainty estimation
  • Explanation-stability analysis
  • Model-complexity and compute-cost comparison
  • Exportable scientific model-decision reports
  • Notebook and research-repository integration
  • Classroom modules for responsible AI education

Our goal is not to discourage students and scientists from using advanced models. Our goal is to help them determine when those models are justified, how they should be validated, and whether additional accuracy is worth the loss of interpretability, reliability, or scientific applicability.

WISE-XAI does not reject advanced AI. It helps users determine when advanced AI is scientifically justified.

Built With

Share this project:

Updates