Shiftglass

Actual paired test: same foreground, changed backdrop. Synthetic accuracy falls from 192/192 to 0/192.
Actual balanced retraining: both environments reach 192/192 correct on this toy task; small, uncalibrated margins remain visible.

Inspiration

A good test score can hide a brittle shortcut. I wanted the first lesson in model evaluation to be something a person could see and change: the same shape, with the same label, suddenly receiving a different answer because its background changed.

What it does

Shiftglass is an interactive shortcut-learning laboratory. Train a classifier on synthetic upright and crossbar shapes whose backdrops correlate with their labels. Pick a held-out specimen and swap only its background. Inspect the actual model score, group performance and paired prediction flips.

Then repair the training distribution. Each training shape appears against both backgrounds, a new model trains from scratch, and the same held-out specimens test the result. Before/after controls keep the difference visible. Exporting a recipe lets another person regenerate the data and retrain both models; replay discards supplied scores instead of trusting them.

At default seed 42, the first model classifies 192/192 original-environment specimens correctly and 0/192 after paired backdrop swaps. Balanced training gives 192/192 in both environments, with paired flips falling from 192 to zero. These are measured outcomes on an intentionally simple synthetic task. The scores are uncalibrated and the balanced model's margins can be small; this is not a robustness guarantee.

How I built it

Original JavaScript implements regularized logistic regression over 768 normalized RGB features, with 120 full-batch gradient updates. Inference receives pixels, not label or backdrop metadata. Canvas shows the same quantized pixels that enter the model. Training yields to the browser so progress remains visible. The interface, image generator and verifier run without an account, installation or external inference API.

Challenges and accomplishments

The intervention had to change only the claimed factor. Tests check that foreground pixels, texture, shape and label remain fixed, that held-out examples never enter training, and that analytical gradients match finite differences. Fifteen local tests pass, including deterministic retraining and malformed-recipe rejection. A separate verifier reproduces the model results and weight hashes. No hosted CI or learner-study result is claimed.

What I learned and what is next

An improved aggregate score is still only part of the story. Paired examples, subgroup scores and honest uncertainty make the result easier to examine. Next I would test whether learners can transfer this reasoning to other visual factors and less artificial datasets. That educational benefit remains a hypothesis, not a measured outcome.

Originality and credits

Shiftglass was created from scratch on September 21, 2026 specifically for BunnieX. The synthetic images, code, interface and experiments are original. OpenAI Codex substantially assisted design, implementation, testing and writing. The actual runtime model is the local classifier. Code is MIT licensed, with system fonts and no third-party runtime dependencies. No previous hackathon entry or private dataset was reused.