-
-
MaskLab logo
Inspiration
The same AI model can answer differently depending on who we ask it to be—or what we tell it we already believe. A confident user, a professional persona, or a misleading suggestion can influence an answer that should remain objectively correct.
We built MaskLab out of an interest in AI interpretability and reproducible research. We wanted a practical way to investigate these behavioral changes, with controlled comparisons and evidence people can inspect.
What it does
MaskLab is a behavioral research workbench for studying how personas and user framing affect language model answers.
Researchers can compare a neutral prompt against different personas, introduce correct or incorrect user beliefs, and examine changes in correctness, consistency, and agreement with false suggestions. Paired responses make individual changes visible, while aggregate results help reveal patterns across questions and repeated trials.
MaskLab studies observable behavior. Its results support specific conclusions about the experiments performed, rather than claims about a model’s internal reasoning.
How we built it
We designed MaskLab around a controlled experiment pipeline: define a dataset, configure independent experimental conditions, preserve the exact prompts, collect responses, and evaluate them against known answers.
The stack combines a React and TypeScript interface with a Python FastAPI backend. SQLite stores experiment records, while Python handles deterministic scoring and statistical analysis. Nebius Token Factory provides inference for NVIDIA Nemotron models.
We used our Agent Control Plane to organize specification-driven development, connecting requirements, implementation tasks, and acceptance criteria.
Challenges we ran into
The hardest design challenge was separating an interesting response change from a meaningful research finding. An answer can change without becoming incorrect, and agreement with a false suggestion requires more careful interpretation than a simple disagreement count.
We also had to account for invalid responses, interrupted requests, repeated trials, and small datasets. These cases need to remain visible so that a clean-looking chart does not hide uncertainty or failures.
Another challenge was keeping the scope achievable while preserving the foundations of credible research: reproducibility, inspectable evidence, explicit scoring rules, and controlled comparisons.
Accomplishments that we're proud of
We’re proud of turning a broad question about AI behavior into a concrete, testable experimental design.
MaskLab brings together independent persona and belief conditions, objective answer scoring, paired response inspection, and traceable experiment records. We also made scientific limitations part of the product design, including distinguishing synthetic test fixtures from a representative benchmark.
What we learned
Evaluating AI requires as much care in experimental design as in implementation. Prompt wording, question selection, scoring rules, and missing responses can all affect the conclusions.
We learned to treat reproducibility as a product feature. Preserving the exact conditions and underlying evidence makes a result more useful than a score alone.
We also learned that behavioral evaluation offers a practical starting point for investigating models through hosted APIs, while requiring clear boundaries around what those observations can est
Built With
- fastapi
- httpx
- mcp
- nebius
- pydantic
- python
- react
- sqlite
- typescript
- uvicorn
- vite
