Safety Evaluation Scorecard for a Legal AI Assistant

DS6051, Decoding Large Language Models — Hackathon Final Report July 8, 2026

Team: Hudson Noyes, Ryan Dallas, Lino De Ros, Jackson Kennedy, Nathan Todd

Code: github.com/nathanctodd/legal-llm-metrics Devpost: devpost.com/software/uva-decoding-llms-project Report: Google Doc Video: Google Drive

1. Project Overview

This project treats a general-purpose open-weight language model as if it had been built and deployed as a consumer-facing legal AI assistant, and asks whether it is safe to deploy in that role. We evaluate google/gemma-4-E2B (base) and google/gemma-4-E2B-it (instruction-tuned) against a five-metric safety scorecard built for the legal domain: factual consistency, sycophancy, calibration, cross-lingual safety distribution shift, and citation accuracy. This is an evaluation exercise only; no model was trained or fine-tuned. Most metrics run on a fixed 100-question subset of MMLU professional_law translated into Spanish and Basque; judge-scored metrics use the 64-question Law slice of TruthfulQA.

Three findings lead the results. First, instruction tuning helps English competence but not safety: accuracy rises from 39% to 48% while expected calibration error rises from 0.093 to 0.410. Second, the instruction-tuned model is highly deferential under social pressure, changing its answer 80% of the time when a user confidently asserts a wrong one. Third, its capability collapses in Basque, falling from 48% in English to 31% (Language Parity Index 0.646), below even the base model on identical items. Both models stay under 50% accuracy in every language; neither is close to deployable.

2. Scorecard

Metric	Result	Why it needs to be included for this use case	How it's measured	Limitations
Factual consistency (final accuracy)	Gemma-4-E2B: 33% Gemma-4-E2B-it: 42% (best)	Whether the model still gave the correct answer after user pressure. An assistant that contradicts itself within a conversation cannot be relied on.	Automated: we compare the model's final selected answer letters against the correct answer.	We are mainly evaluating the model's final selected answer rather than the quality of its full explanation.
Sycophancy (gold-to-false flip rate)	Gemma-4-E2B: 7% (best) Gemma-4-E2B-it: 12%	How often the model started correct but switched to the user's wrong answer. Clients push back and misstate the law; a safe assistant must not give in.	Automated: record initial answer, inject a confident false user answer, request a final answer, measure flips.	One pressure phrasing and one turn; misses hedging that stops short of a full flip.
Calibration	ECE = 0.348 overall (n=64); worst on adversarial prompts: ECE 0.497, accuracy 37.9% but mean confidence 87.6% — vs. non-adversarial ECE 0.243, accuracy 71.4%	The assistant must know when to defer to a human lawyer; overconfident wrong answers are the dangerous failure mode.	Automated: self-reported verbal confidence (1–10, elicited in-prompt) vs. LLM-judge truthfulness verdict, summarized as ECE over 64 TruthfulQA-Law questions.	Small n=64; confidence is self-reported prose, not a token probability; possible judge error; adversarial prompts inflate calibration difficulty.
Cross-lingual shift	LPI: Spanish 0.938, Basque 0.646 (base: 0.949, 0.974)	Non-English speakers rely heavily on legal aid; capability that silently collapses in a low-resource language is a safety failure for those users.	Automated: the same 100 items paired across EN/ES/EU by id; LPI = accuracy(language) / accuracy(English).	Competence gap, not a refusal gap; translation artifacts may understate the shift (Section 6).
Citation accuracy	Attach rate 43/102 claims; all 9 verifiable citations real; corpus recall 0.42	Fabricated case citations are the canonical legal-AI failure; attorneys have been sanctioned for filing AI-invented cases.	Measured across four conditions: a citation's existence (mitigating ghost citations), fidelity of quotations pulled directly from a citation, precision of each citation, and recall across all citations for a given response.	Convoluted process, can be computationally expensive.

Table 1. Complete safety scorecard for gemma-4-E2B-it evaluated as a legal assistant, in the assignment's required format; base-model results in parentheses.

3. Evaluation Methodology

3.1 Dataset and languages

All automated metrics use MMLU professional_law, a bar-exam-style multiple-choice benchmark with ground-truth labels. The primary benchmark is a fixed, reproducible 100-question subset of the test split (shuffled with seed 42, first 100 items); the full 1,534-question English split serves as a robustness reference. The subset was translated into Spanish and standard Basque with a consistency-controlled LLM translation pipeline using a fixed legal-terminology glossary, then validated at merge (paired ids, identical answer indices, harmonized terminology). Common-law terms with no civil-law equivalent carry the English in parentheses on first use; caveats are in data/TRANSLATION_NOTES.md. Judge-scored metrics (citation accuracy and a verbalized-confidence calibration study) use the 64-question Law slice of TruthfulQA, an adversarial benchmark with reference answers, built to elicit confident falsehoods.

3.2 Models and inference

We evaluate google/gemma-4-E2B (base, plain-text prompts) and google/gemma-4-E2B-it (chat-formatted prompts via its chat template). Both models were run with greedy decoding on a single 24GB GPU via Rivanna's interactive partition. For multiple-choice scoring, we read the model's logits over the four answer-letter tokens directly rather than parsing generated text, taking the argmax as the predicted answer and the softmax probability of that letter as the confidence score. For open-ended citation/truthfulness evaluation, responses were generated with maximum tokens set to 256, and confidence was elicited via an explicit "Confidence: 1–10" field in the prompt, parsed from the model's free-text output. Code is under code/ and calibration_eval.ipynb.

3.3 Accuracy, calibration, and cross-lingual shift

Each model answers all 100 questions per language; predictions are scored against gold labels and per-answer confidence is summarized as expected calibration error (ECE). Cross-lingual shift compares per-language accuracy on the paired items via the Language Parity Index (Section 7).

3.4 Factual consistency and sycophancy

We use a scripted multi-turn evaluation: the model answers a question (initial answer), is challenged with a known-incorrect answer framed as a confident user claim, and is then asked for a final answer only. Because MMLU has gold labels, scoring is fully automated, yielding initial accuracy, final accuracy, challenge change rate, false adoption rate, and gold-to-false flip rate (started correct, switched to the user's wrong answer).

3.5 Citation accuracy

Additional code was injected into the judge module to evaluate citation accuracy. In essence, three additional modules (one for class definitions, a second for helper functions, and a third for major functions — for code cleanliness) run a series of functions evaluating identified cornerstones of citation accuracy: a citation's existence, precision of the citation, the fidelity of quotes directly pulled from references in a citation, and the recall of the aggregation of citations.

3.6 LLM-as-judge

For open-ended outputs, a separate judge grades correctness so the model under test never grades itself: Qwen2.5-7B-Instruct receives the question, the model's answer, and TruthfulQA's reference correct and incorrect answers, and returns a truthful verdict read from Yes/No token logits. In the companion calibration study on this slice, the model self-reports a 1–10 confidence per answer, scored against judge-graded accuracy as ECE, with the overall value at 0.35. Per the assignment we compared judges: an earlier run used Qwen2.5-3B-Instruct, and moving to the 7B judge produced a large jump in graded accuracy from ~4% to ~65%, implying the LLM-as-judge is semi-reliable and model-dependent.

4. Results

Model	English (n=100)	Spanish (n=100)	Basque (n=100)	Full English set (n=1,534)
Gemma-4-E2B (base)	39%	37%	38%	40.9%
Gemma-4-E2B-it	48%	45%	31%	45.6%

Table 2. Multiple-choice accuracy on MMLU professional_law. Chance is 25%.

Model	Spanish LPI	Basque LPI
Gemma-4-E2B (base)	0.949	0.974
Gemma-4-E2B-it	0.938	0.646

Table 3. LPI = accuracy(language) / accuracy(English), English baseline = 1.000. Paired 100-item subset.

Model	English	Spanish	Basque	Full English set
Gemma-4-E2B (base)	0.093	0.068	0.085	0.042
Gemma-4-E2B-it	0.410	0.413	0.488	0.404

Table 4. Expected calibration error (ECE); lower is better. Plots in `results/Nathan_Results/Plots_/`.*

Model	Initial accuracy	Final accuracy	Contradiction rate	Challenge change rate	Gold-to-false flip
Gemma-4-E2B (base)	36%	33%	19%	22%	7%
Gemma-4-E2B-it	44%	41%	6%	80%	12%

Table 5. Multi-turn consistency and sycophancy on the 100-item English subset.

Notes: Initial accuracy (44%) differs from Table 2 (48%) due to pipeline differences.

5. Discussion of Results

5.1 Cross-lingual shift

The instruction-tuned model loses a third of its English capability in Basque (48% to 31%, LPI 0.646), landing below the base model on identical items, while Spanish holds up (45%, LPI 0.938): instruction-tuning gains concentrate in high-resource languages, so a multilingual deployment would silently serve Basque speakers a categorically worse product. The base model's near-flat profile (39/37/38%) is uniform weakness near the 25% chance floor, not robustness.

5.2 Calibration

Overall calibration is poor (ECE = 0.348, Brier = 0.352) and the failure is not uniform — it concentrates exactly where it matters most. On adversarial prompts, accuracy drops to 37.9% while mean stated confidence stays at 87.6%, producing the widest overconfidence gap in the scorecard (0.497) and the highest per-group ECE (0.497). On non-adversarial prompts the model is still overconfident but far less dangerously so (accuracy 71.4%, ECE 0.243, gap 0.100). The bin-level breakdown makes the risk concrete: 36 of the 64 answers fall in the highest confidence bin (0.8–1.0), yet only 52.8% of those are actually correct — meaning most of the model's most confident answers are barely better than a coin flip. Since this confidence is self-reported prose rather than a token-level probability, a downstream system cannot use it as a reliable signal for when to defer to a human lawyer without independent recalibration.

5.3 Sycophancy and consistency

For the factual consistency/sycophancy metric, we used a scripted legal multiple-choice pipeline: each model answered 100 MMLU professional law questions, was challenged with a confidently stated but incorrect user answer, and was then asked for a final answer. This measures whether a legal assistant can maintain a grounded answer under user pressure rather than agreeing with a false claim. We use final accuracy as the factual consistency measure and gold-to-false flip rate as the sycophancy measure, meaning cases where the model was initially correct but switched to the user's wrong answer. Gemma-4-E2B achieved 33% final accuracy with a 7% gold-to-false flip rate, while Gemma-4-E2B-it achieved 42% final accuracy with a 12% gold-to-false flip rate. This suggests the instruction-tuned model was more accurate after user pressure, but still showed some susceptibility to false user correction. The metric is a controlled test of legal answer stability, not a complete measure of legal assistant safety, because it uses multiple-choice questions and tests one specific form of sycophancy rather than open-ended legal reasoning or real-world legal advice.

5.4 Citation behavior

The model attached a citation to only 43 of 102 claims (corpus recall 0.42); all nine verifiable citations resolved to real authorities (four case-law, five U.S.C.), so the dominant failure at this scale is under-citation rather than fabricated authority. The other 34 named instruments are unverifiable by design, and nine verifiable citations are too few to certify the model never fabricates case law.

6. Limitations

Multiple-choice bar-exam questions measure knowledge recall, not open-ended advice or drafting; at 100 items per language, few-point differences are within noise (the Basque collapse is far larger).
Translations were LLM-generated with glossary control and validation but not reviewed by native speakers; caveats in data/TRANSLATION_NOTES.md likely make the Basque estimate conservative.
Sycophancy uses one scripted pressure pattern, and ECE uses answer-token confidence rather than the verbalized certainty users see.
Judge-graded metrics inherit judge error: graded accuracy jumped materially between Qwen2.5-3B and 7B judges, so results are judge-dependent; on TruthfulQA, self-reported confidence clusters on round values and n = 64 makes ECE noisy.

7. Proposed Metric (Bonus): Language Parity Index

We propose the Language Parity Index, LPI(language) = accuracy(language) / accuracy(English), as a reusable metric for multilingual deployment: a dimensionless number encoding what fraction of the English-language product non-English users actually get. Its failure mode appears in our own results: a uniformly weak model shows high parity (base model, 0.974 in Basque while barely beating chance), so LPI must be reported with absolute accuracy and the chance baseline. A reasonable gate: LPI at or above 0.9 for every supported language, with English accuracy above a competence floor.

8. Conclusion and Deployment Recommendation

Treated as a legal AI assistant, gemma-4-E2B-it fails the scorecard on three independent axes: unreliable confidence (ECE 0.410), capitulation to confidently wrong users (80% challenge change rate), and silent capability collapse for Basque speakers (LPI 0.646). Neither variant should be deployed in any language. Any successor system needs three launch gates: recalibrated confidence before escalation logic, adversarial multi-turn pressure testing, and per-language parity requirements — making each added language a safety decision, not a localization checkbox.

NB: Claude was used as an assistant in code generation and deliverable production.

Built With

gemma
python

Submitted to

DS6051 Decoding LLMs Hackathon
- Winner 1st Place

Created by

I was primary architect of the citation accuracy modules, analyzing the interface between the provided code for inference and judging, and connecting the two segments for final output.

ryan-Dallas Dallas
I worked on the sycophancy and factual consistency evaluation metrics. I used Rivanna to analyze Gemma-4-E2B and Gemma-4-E2B-it's responses and wrote up that section of the report.

Lino De Ros
Hosted Python notebook in Rivanna HPC and ran all jobs. Collected and gathered information, data, and stats from all jobs. Compiled code across team members.

Nathan Todd
I was primarily responsible for finding datasets and designing the calibration metric. I also oversaw the LLM-as-judge implementation.

jacksonpkennedy
I built the evaluation dataset behind our cross-lingual metric: a fixed 100-question subset of MMLU professional law, translated into Spanish and Basque through a glossary-controlled pipeline and validated so every question stays paired across all three languages. I also assembled the final report and the slide visual, pulling the team's results into the scorecard, discussion, and deployment recommendation.

Hudsonnoyes