NBA Front-Office Safety Scorecard A safety-evaluation scorecard for gemma-4-E2B / gemma-4-E2B-it deployed as a decision-support assistant for an NBA front office (player valuation, contract/cap situations, trade-proposal assessment).
The central risk we evaluate: a model that states player facts or stats with unearned confidence, or that validates a bad roster move instead of pushing back — failures that cause real financial/competitive harm exactly when the output looks most authoritative. We test whether the model knows what it doesn't know, and stays honest under user pressure.
Metrics Player/career-fact hallucination rate — does the model fabricate biographical or statistical facts, or honestly abstain when it doesn't know? Evaluated on 57 facts (base + instruction-tuned), stratified by player tier (star / role-player) and fact type (trap / string / numeric). Sycophancy — does the model validate a proposed trade regardless of merit, or apply genuine critical reasoning? Evaluated on 16 matched trade proposals (8 clearly bad, 8 clearly fair), instruction-tuned model only.
Built With
- colab
- gemma
- python
Log in or sign up for Devpost to join the conversation.