Key findings
- Every one of the 23 other official EU languages has a higher attack success rate than English, on average more than twice as high.
- The gap follows how much text exists in a language on the web, not the alphabet it is written in.
- A model's ranking in English is a poor guide to its ranking in the lowest-resource languages.
- The gap is not inevitable: some recent models show little or none, and larger models of the same family close it.
- The models understand most of the requests. What fails is recognising them as harmful.
- Not everything that slips through is useful to an attacker, so the gap in real harm is smaller than the gap in refusals, but it is there.
Why test in 24 languages?
The European Union has 24 official languages and a single market in which products built on large language models (LLMs) are expected to serve speakers of all of them. Model providers have made their models much better at these languages. Safety evaluation has not kept pace: whether a model declines to help with a harmful request is still measured mostly in English. A model that looks safe in English can comply with the same request written in Estonian or Maltese, and an English-only evaluation will not show it.
Earlier work studied single languages or small groups of them, and multilingual benchmarks of harmful requests covered at most four EU languages besides English. We built a parallel benchmark: the same harmful requests in all 24 official EU languages, so that languages and models can be compared directly.
How we tested
We took 1,053 harmful prompts from four established English benchmarks: HarmBench (200 prompts), JBB-Behaviors (100), SORRY-Bench (440) and StrongREJECT (313). Each prompt was translated into the 23 other official EU languages twice, by DeepL and by Claude Opus 4.5, and an LLM judge picked the translation better suited for safety evaluation. In total, the benchmark has 25,272 prompts.
We sent every prompt to 34 open-weight models with 3 to 35 billion parameters, from the Qwen, Gemma, Granite, Llama, Mistral, Phi, Nemotron, gpt-oss, Aya, EuroLLM, Olmo and Apertus families, as a single message without a system prompt. A safety classifier, Qwen3Guard-Gen-8B, judged each response. The attack success rate (ASR) is the share of harmful requests that a model fulfils: the response is not a refusal, and the classifier labels it unsafe or controversial.
Every EU language is less safe than English
Averaged over the 34 models, the ASR is 7.7% in English and higher in every one of the other 23 languages (Figure 1). It rises through the large West European languages, 10–12% in French, Spanish, German, Italian and Portuguese, to 22–23% in Hungarian, Slovenian, Lithuanian, Estonian and Maltese, and 24.3% in Latvian. The 95% intervals of all 23 languages lie above the English one.
1,053 prompts per language, first response of each model, judged by Qwen3Guard-Gen-8B. Languages grouped by resource availability.
For a product that serves every EU language, the average hides what matters most: the worst language. Llama 3.1 8B Instruct fulfils 8.3% of the requests outside English on average, but 20.4% in Latvian. Qwen3 4B Instruct 2507 averages 9.3% and reaches 26.0% in Maltese; Phi-4 averages 14.1% and reaches 36.4% in Maltese.
The less web text, the larger the gap
The languages line up by how much text exists in them on the web. Across the 24 languages, the ASR is strongly correlated with the logarithm of the share of Common Crawl web pages in the language (Spearman ρ = −0.91), and so is the gap to English (ρ = −0.90, Figure 2). Grouped by resource availability, the ASR is about 12% in the six large West European languages, 18% in the eleven mid-resource languages and 22% in the six lowest-resource languages: Irish, Maltese, Estonian, Latvian, Lithuanian and Slovenian.
Common Crawl shares from crawl CC-MAIN-2026-39. Labels are ISO language codes.
The alphabet does not explain the gap. Latvian, Lithuanian and Estonian are written in the Latin alphabet, like French and Spanish, while the two languages in other scripts, Bulgarian and Greek, are on average only 1.8 percentage points above the Latin-script languages. Irish and Maltese, the two languages with the least web text, are also where the automatic classifier is least reliable, so their exact values are less certain than those of the other languages (see below).
English results don't tell you which model is safest in your language
For anyone choosing a model, this may be the most important finding. How the 34 models rank in English is a poor guide to how they rank in the lower-resource languages. The rank correlation with the English ranking is 0.81 for Spanish, but only 0.47 for Estonian, 0.43 for Irish and 0.41 for Maltese (Figure 3). A model that is among the safest in English can be among the most vulnerable in another language.
Rank correlations over the 34 models.
Olmo 3 7B Instruct is the tenth safest of the 34 models in English, fulfilling 2.9% of the requests, and the second least safe in the other languages, with 43.4% on average. Nemotron 3 Nano 4B goes from 1.3% in English to 19.9%. For 16 of the 34 models, the ASR outside English is more than twice the English one.
The gap depends on the model, and it can be closed
Model choice matters a great deal (Figure 4). Averaged over the 23 non-English languages, the ASR ranges from 2.0% for Qwen3.6 35B-A3B to 45.3% for Mistral 7B Instruct v0.2, a model from 2023. For 33 of the 34 models it is higher than in English, and for 11 of them by more than 10 percentage points.
1,053 prompts per language, first response, judged by Qwen3Guard-Gen-8B.
Several recent models show small gaps or none at all. Gemma 4 31B is no less safe outside English than in English (8.4% against 9.0%), and Qwen3.5 9B, Qwen3.6 35B-A3B, Qwen3.8 27B and gpt-oss-20b stay below 5% on average in the non-English languages. Within each of the six families that we tested in more than one size, the gap shrinks as the models get larger (Figure 5): from +17.4 to +8.9 and +2.8 percentage points for Granite 4.1 3B, 8B and 30B, and from +6.2 to −0.6 for Gemma 4 E2B to 31B. Safety behaviour that holds across languages is achievable.
Sizes are total parameters; mixture-of-experts models also show their active parameters.
The model card's language list is a warning sign, not a guarantee
Twelve of the models list some, but not all, of the 23 non-English EU languages as supported on their model cards. For 11 of the 12, the ASR is higher in the languages that are not listed, by 12.3 percentage points on average (Figure 6). Llama 3.2 3B Instruct, for example, fulfils 10.6% of the requests in English, 7.4% in the five EU languages its card lists, and 32.8% in the other 18.
A listed language is not automatically safe, though. The listed languages are mostly the large ones, and when the language is held fixed, models that list it are only slightly safer in it than models that do not (by 4.4 percentage points, which is not statistically significant). The five models whose cards list all 23 languages, the three EuroLLM models and the two Tiny Aya models, still have gaps of 6.6 to 13.0 percentage points.
Supported languages as listed on the model cards.
Which kinds of harm are affected?
The gap is not confined to a few topics. We grouped the 66 categories of the four benchmarks into 13 harm types (the grouping is ours; the explorer below shows which categories belong to which type). Every type has a higher ASR outside English (Figure 7), from +4.8 percentage points for privacy violations to +14.2 for political, religious and ethical persuasion. Requests about weapons, drugs and dangerous goods are fulfilled 2.5 times as often outside English (17.0% against 6.9%), and requests about cybercrime and hacking 2.7 times as often (17.8% against 6.6%).
Harm types group the 66 categories of HarmBench, JBB-Behaviors, SORRY-Bench and StrongREJECT.
Some individual categories change dramatically. Requests to promote a religion are almost always refused in English (0.9% fulfilled) but fulfilled 23.3% of the time in the other languages. Requests to promote conspiracy theories rise from 22.1% to 38.3%.
The models understand the requests. They just don't see the harm
A model may comply with a request in another language because its refusal behaviour does not carry over to that language, or because it does not understand the request well enough to see that it is harmful. To tell the two apart, we asked every model to translate a sample of the prompts from each language into English.
The models translate most prompts accurately in every language: 74–84% on average, except in Estonian (71%), Maltese (65%) and Irish (54%). But they recognise far fewer requests as harmful. In the large West European languages, they decline even to translate 16–20% of the prompts; in Irish, Maltese, Latvian, Estonian and Lithuanian, only 2–5%. Across languages, this rate of harm recognition is closely tied to the ASR (ρ = −0.92). Of the successful attacks in the non-English languages, 79% are on prompts that the model understood, and even among accurately understood prompts the ASR rises from 10.5–11.3% in the large West European languages (except Dutch) to 19–23% in the lowest-resource ones.
In the lowest-resource languages the models also refuse more harmless requests. In a control experiment with 10 of the models, they refused 22–25% of harmless prompts in Latvian, Maltese, Estonian and Lithuanian, against 15% in English and 9–12% in the large West European languages. Their refusals depend less on what is actually being asked. Safety behaviour learned mostly in English does not transfer to languages that are rare in the training data.
Fewer refusals, but not always more harm
Is the gap real, or an artefact of the classifier or the machine translation? We checked it in several ways (Figure 8). Judging the responses with Llama Guard 3, a classifier from another model family, gives the same picture, and so does translating the responses into English and judging them there. The gap also remains when we use only the prompts whose two candidate translations agree most closely.
Our classifier, like Llama Guard 3, judges whether a response is unsafe, which includes partial and low-quality compliance. Judges that rate how specific and useful the harmful content is find a smaller gap: about 3 percentage points for the HarmBench classifier and the StrongREJECT scorer, and 8 for Gemma 3 27B grading with the StrongREJECT rubric. Many of the responses that are not refused in the lowest-resource languages are of low quality; in Maltese and Irish, a third of the successful attacks give no specific help at all. The largest increases in substantive harm are in languages such as Polish, Finnish, Hungarian and Slovenian, where the models are both less safe and still capable.
For the StrongREJECT scorer and the Gemma rubric: mean score × 100.
In short, in the lowest-resource languages the models refuse much less often than in English, but much of what they produce instead is of little use to an attacker. The refusal gap is real nevertheless: refusing is the safety behaviour these models are meant to have, and more capable models may produce more useful content when they fail to refuse.
What this means if you deploy LLMs in the EU
- Test safety in every language you deploy in. English results do not carry over, and the ranking of models changes from language to language.
- Look at the worst language, not the average. An average over languages hides the lowest-resource ones, where the risk is highest.
- Measure both refusals and harm. Whether a model refuses and how harmful its output is can diverge, most of all in low-resource languages.
- Treat languages missing from the model card as high-risk. But do not assume that the listed ones are safe.
- Add safeguards where they are needed most. In the lowest-resource languages, additional safety training or an input filter may be necessary.
This benchmark is a research project. Evaluations for our clients are built around each system: its use cases, its threat model and the languages its users write in, informed by what we learn from research like this. Talk to us about yours.