Beyond Refusal: Multilingual Content Moderation for Gemma

Project Story and Overview

Inspiration You may have seen stories of chatbots responding to vulnerable or hostile users in ways that escalate harm instead of reducing it, such as encouraging self-destructive behavior, reinforcing hateful ideas, or complying with abusive requests. These failures show why content moderation cannot just be an afterthought.

Harmful LLM prompts involving discrimination, harassment, bullying, and self-harm can appear in many linguistic and cultural contexts, and safety behavior that works well in one language may not transfer cleanly to another. For this hackathon, we wanted to understand whether instruction tuning makes Gemma meaningfully safer when responding to multilingual red-teaming prompts, and whether that safety holds up outside of English.

Our project focuses on that problem by evaluating how well Gemma handles multilingual red-teaming prompts involving discrimination, self-harm, bullying, and harassment, and whether instruction tuning helps the model respond more safely.

What it Does Our evaluation used a small multilingual red-teaming dataset derived from the CohereLabs Aya Red-teaming corpus. We selected prompts from three safety-relevant categories: Discrimination & Injustice, Bullying & Harassment, and Self-Harm. The final evaluation set included prompts in English, Spanish, and Serbian, allowing us to compare Gemma-base and Gemma-IT across different languages and harm categories.

For each prompt, both Gemma models generated a response. We then used an LLM-as-judge framework to evaluate each response on three metrics: Harm Recognition, Refusal Behavior, and Safe Steering. Three independent judge models scored every response using the same rubric, assigning scores from 1 to 5 for each metric. We averaged the judges' scores and then aggregated the results by model, language, and harm category to produce our final safety scorecard.

Building the Pipeline We built a reproducible Python evaluation pipeline using Hugging Face datasets and Transformers. First, we created a deterministic sampling script to pull 50 prompts per language from the Aya red-teaming dataset, stratified across the three selected harm categories.

The dataset was built from the CohereLabs Aya Red-teaming dataset, using 150 multilingual prompts designed to test model safety behavior. We sampled 50 prompts each in English, Spanish, and Serbian, with examples stratified across three harm categories: Discrimination & Injustice, Self-Harm, and Bullying & Harassment. The sampling process was deterministic, using a fixed seed so the evaluation set can be reproduced. Each prompt includes metadata such as language and harm category, allowing us to compare how Gemma base and Gemma instruction-tuned models respond to the same safety challenges across different languages and types of harmful content.

We then ran inference on both google/gemma-4-E2B and google/gemma-4-E2B-it. The base model received raw text prompts, while the instruction-tuned model used a chat template with a simple helpful-assistant system prompt. Inference and judging were run on a Google Colab A100 GPU session.

For evaluation, we designed a shared LLM-as-judge rubric and applied it blindly: the judges never saw which model produced a response. Three judge models scored each response: Gemma IT, Qwen, and Mistral. We aggregated their scores into a final scorecard and used a notebook to generate charts and judge-agreement analysis.

Challenges LLM judges can be biased, inconsistent, or incorrect, particularly for borderline safety cases or non-English prompts. Additionally, one of the judge models was Gemma-IT, which may have introduced self-evaluation bias (Gemma-IT evaluated its own responses slightly higher than other judge models). Next, our dataset is relatively small and limited to three harm categories, so the results should be interpreted as a focused comparison rather than a comprehensive measure of overall model safety. Finally, the dataset could be expanded to include a range of safe to harmful prompts, so that the models could be additionally evaluated on their responses across this range.

One major challenge was designing a fair rubric. We did not want judges to reward generic refusals in every situation; the response needed to be appropriate to the actual prompt. That meant the rubric had to evaluate safety accuracy, not just whether the model sounded cautious.

Another challenge was multilingual evaluation. English, Spanish, and Serbian prompts can carry different context, phrasing, and cultural cues. The judges had to evaluate responses in the original language without translation. This made judge consistency an important limitation to track.

We also had to manage compute resources carefully. Running two subject models and three judge models over hundreds of responses required batching, GPU memory management, reproducible decoding, and scripts that could save progress along the way. We made the decision to use the A100 GPU session on Google Colab instead of the given Rivanna instance so as to not run into any problems on this front.

Accomplishments We are proud of building an end-to-end pipeline for multilingual safety evaluation. Our pipeline goes from dataset construction and sampling to inference, judging, aggregation, and visualization. The project is reproducible, with pinned dependencies, deterministic sampling, and clear scripts for each stage. We are happy with our clear result that instruction training produced dramatically improved safety behavior. Gemma IT scored 4.97 overall compared with 2.37 for the base model. The instruction-tuned model consistently recognized harmful prompts, refused unsafe requests, and redirected toward safer alternatives across every tested language and category. Finally, we are proud that our evaluation went beyond a single aggregate score. We examined performance by language, category, and judge agreement, which helped reveal both the strength of the instruction-tuned model and the uncertainty involved in LLM-based safety evaluation.

Takeaways We learned that instruction tuning had a significant impact on safety behavior in this evaluation. Gemma-IT consistently recognized harmful prompts, refused unsafe requests, and redirected users toward safer alternatives, while Gemma-base often treated harmful prompts as ordinary text-completion tasks. This demonstrates how much instruction tuning and post-training alignment influence the safety of user-facing language models.

We also learned that multilingual safety evaluation is essential. Even when a model performs well overall, its safety behavior can vary across languages and harm categories, particularly with highest scores for English and slightly lower scores for prompts in Serbian. Finally, our LLM-as-judge framework proved to be an efficient way to evaluate many prompt-response pairs at scale. However, it also highlighted that automated safety scores should be treated as evidence rather than ground truth, since judge models can be biased or inconsistent.

What's Next for Multilingual Content Moderation? We’re interested in expanding the dataset across a range of safe to harmful prompts to evaluate responses across a mix of harm. For example, do LLMs effectively alter their responses for neutral or positive prompts that reference ethnic groups? The “Refusal Behavior” metric is important in this next step, since unintentionally refusing benign prompts would be inconvenient for users.

Longer term, this project could become a lightweight multilingual safety evaluation framework that developers can use to compare model behavior before deploying AI systems.

Discussion of Results

EVALUATION We evaluated 150 prompts drawn from the CohereLabs Aya Red-Teaming dataset, restricted to three harm categories (Discrimination & Injustice, Bullying & Harassment, Self-Harm) across three languages (English, Spanish, Serbian). Both Gemma-base and Gemma-Instruction-Tuned generated a response to every prompt, and each response was scored by three separate LLM judges (Gemma-4-E2B-it, Qwen3-4B-Instruct, Ministral-8B-Instruct) using a single shared rubric. Judging was blind as no judge was told which model produced the response. Each judge returned three 1–5 accuracy scores across the following metrics: Harm Recognition, Refusal, and Safe Steering. Critically, these are accuracy metrics, not raw safety metrics. The rubric instructs the judge to first classify the prompt as harmful or benign, then reward behavior appropriate to that classification. In other words, refusing a benign prompt scores just as low as complying with a harmful one. This design guards against widespread refusal, but a limitation is that this scoring is conditional on a judge classifying the prompt correctly. If a prompt is misclassified by the judge, all three metrics for that response are corrupted.

RESULTS

Model	Harm Recognition	Refusal	Safe Steering	Overall
Gemma Base	2.09	2.64	2.38	2.37
Gemma IT	4.94	4.99	4.98	4.97

Gemma-IT far outscored Gemma-base on every metric, in every language, and in every harm category (overall 4.97 vs. 2.37). The base model's weakest metric was Harm Recognition (2.09) and its "best" score was Refusal (2.64) which reflects incoherent responses. The safety gap across languages appeared only in the base model, which degraded from 2.76 in English to 2.25 in Spanish and 2.09 in Serbian. The Gemma Instruction Tuned (IT) model performed well in all three languages (4.96–4.98). All three judges agreed on IT responses (mean per-response SD=0.04) but diverged on the base model responses (0.95), with Gemma-as-judge the most lenient (3.07) and Mistral the harshest (1.76).

Across categories, languages, and metrics, the judges gave the IT model a near perfect score. This does not mean that the model is flawless, but rather it means that the judges rarely found anything to dock. Fixing this would require a more comprehensive and harsher grading rubric.

LIMITATIONS Our evaluation rests on 150 prompts in three categories and three languages. While this is large enough to show a consistent direction, this set is too small to confirm model safety or support fine-grained per-category claims. Additionally, the judge models are small (2–8B parameters), and their reliability in Spanish and especially Serbian is unvalidated. Finally, Gemma-IT also serves as one of its own judges. While judging is blind, Gemma-IT may have an implicit bias towards its own outputs.

Overall, we can conclude from our analysis that the Instruction Tuned model has more safety guardrails in its generated responses; however, this is not a safety guarantee.

Scorecard

Metric	Result (Base → IT)	Why Included	How Measured	Limitations
Accurate Harm Recognition (1–5)	2.09 → 4.94	Does the model spot harmful speech without flagging benign prompts?	LLM-as-judge (3 judges average)	Requires correct harm classification; Gemma self-judges.
Accurate Refusal (1–5)	2.64 → 4.99	Refuses harmful requests while still helping with benign requests.	LLM-as-judge (3 judges average)	Judges prompt text rather than intent; weaker on non-English prompts.
Accurate Steering (1–5)	2.38 → 4.98	Redirects to a safer, more supportive alternative when warranted.	LLM-as-judge (3 judges average)	Subjective metric; constructive steering is a judgment call.
Overall Safety (1–5)	2.37 → 4.97	One headline comparison between Base and IT.	Mean of the three metrics above.	Simple mean weights all metrics equally; a high score does not guarantee real-world safety.
Judge Agreement (SD)	0.51	Do the three judges agree?	Standard deviation of judge scores.	Agreement does not necessarily imply correctness.
Coverage	900/900 (100%)	Confirms every response was scored.	Automated JSON count.	Only verifies a score exists; not a measure of quality.

Individual Work Recap

Our group worked together throughout the whole hackathon. We all did a lot of work on the project. Some of our specific contributions include Sam Kunitz-Levy and Will Brannock coming up with the content moderation angle (first we thought we would use Tom Hartvigsen’s Toxigen dataset). Then we realized we needed multilingual prompts. Reed researched publicly available datasets, went through prompts, and eventually found and vetted the dataset we used from Cohere. Will Novak worked on the metrics that the LLM judges would use to evaluate the content moderation prompts. Will Novak and Reed both poured over the dataset to identify how the specific prompts would affect the metrics that we chose. They iterated on the metrics design to come up with the right set. Sam helped on this effort, identifying the analysis output and proposing hypotheses that we would test. Eddie and Will Brannock set about designing the pipeline. Eddie figured out how to pull the prompts from Cohere and reengineered the system prompts to make sure the LLM judges effectively evaluated the prompts. Will Brannock created and ran the scripts to run the pipeline, then Eddie put together the results table and contributed to the discussion. Reed aggregated everything into this lovely Devpost you are now reading and thoroughly enjoying.