At 6:45 a.m. Maya Rodriguez reheats the same cup of coffee. She needs a short worksheet on energy transfer for three eighth-grade sections. The English version sits half-finished on her laptop. She opens a general AI assistant, pastes her draft, and asks for a Bengali version that matches it exactly. The output reads smoother in places. One sentence slips in the phrase “as most people agree.” She deletes it before she prints anything.
That small edit is the kind of habit our study on multilingual LLM bias supports. We asked whether the language of a prompt changes how much a commercial model resists attempts to steer its answer. It does. The gap widens in Hindi and Bengali compared with English, especially when the prompt applies pressure.
I am an AI engineer at Accenture and a PhD candidate at IIIT Kalyani researching bias in multilingual large language models. My co-authors and I designed the tests. We wanted to measure whether a model’s willingness to stay neutral depends on the language of the request.
How the test worked
We took three established cognitive-bias benchmarks. We obtained verified translations into Hindi and Bengali. Then we ran the translated benchmark questions in English, Hindi and Bengali across six commercial LLMs.
The total came to 683,616 model responses. We compared neutral-response rates across languages and prompt types. A neutral-response rate is the proportion of evaluated answers classified as neutral rather than following the nudge.
One result concerns confirmation-pattern prompts. These are the ordinary ways people nudge an answer. As invented illustrations, not quotations from the study, a user might add “I think the correct choice is A, don’t you agree?” or “I have already decided B is right. Explain why B must be right.” The study measured how often the model kept its neutral stance anyway.
Verified translations let us compare the benchmarks across three languages, though wording cannot be identical in every respect. The full methods and results appear in the published paper, How prompts manipulate models: measuring cognitive bias vulnerability in commercial LLMs across languages, in Innovations in Systems and Software Engineering (Springer).
What the numbers showed
Neutral-response rates ranged from 74.11 percent to 94.54 percent across models on the same dataset. One model held neutral in roughly three out of four cases. Another stayed neutral in more than nine out of ten.
Confirmation-pattern prompts reduced neutral answers by 19.42 percentage points on average across the tests. Vulnerability was higher in Hindi and Bengali than in English, particularly under manipulative prompts.
These figures describe the six models we tested, under the settings we used. Tools and their settings change. The study recorded what happened with those models at that time.
What the measures mean, and what they do not
A neutral-response rate is the proportion of evaluated answers classified as neutral. It does not say whether an answer was factually correct. A model can stay neutral and still be wrong, or follow a slant and land on the right answer by accident.
The 19.42-percentage-point change is an average across a large set of responses. It does not mean every prompt in Hindi or Bengali will tilt a model. It means that, across the tests, manipulative prompts reduced neutral answers by that amount.
Because the three benchmarks were translated and verified, we could compare like with like. The verified translations made comparisons across the three languages more meaningful, though they cannot make every feature of the wording identical.
Neutrality, accuracy, and translation quality
Three different problems can look the same in a chatbot answer. Neutrality is about stance. Factual accuracy is about whether the statements match reality. Translation quality is about whether the wording carries the same meaning across languages.
A model can produce a fluent Bengali answer that is neutral in tone but factually wrong. It can produce an accurate answer that subtly favors one interpretation. It can also translate a neutral English prompt into Bengali and introduce a slant that was not there before.
Our study measured only the first of these. A neutral-response rate does not tell you whether an answer is factually correct. We tested whether the models resisted pressure to take a side.
Multilingual LLM bias in a science classroom
Picture Maya Rodriguez again. She wants five retrieval questions on thermal energy for her mixed-language class. She writes the prompt in English first.
You are helping an eighth-grade science teacher.
Context: [Students have learned that thermal energy transfers from warmer objects to cooler ones. Cold is not a substance that moves. When the temperature of a substance rises without a phase change, its particles' average kinetic energy increases.]
Goal: [Draft five short retrieval questions plus one short-response item that I will review.]
Constraints: [Use plain language. Stay within the three facts above. Keep tone neutral and factual. Do not editorialize or add opinions.]
Output shape: [Number the questions. Add a separate answer key. Flag any item I must verify.]
She runs the English version. The output looks clean. Then she translates the prompt into Bengali and runs it in the same general AI assistant. She places the two drafts side by side.
In the Bengali version one question contains the clause “as we all know.” She removes it. She checks both answer keys against her own notes. The tool drafted. She checked. She decided what her students would see.
That clause is not proof that the measured effect appeared in her task. It is a reason to check both drafts. Comparing the two versions is a practical response to multilingual LLM bias, not a replication of the study.
How a social studies teacher uses the same check
Frank Bell teaches the same eighth graders. He needs a short explanation of a historical event that students can read in Hindi or English. He opens the tool and adds one sentence to his prompt.
Maintain strict neutrality. Do not favor any single interpretation.
He runs the prompt in English. Then he runs the Hindi version. He scans for any phrase that implies one view is obvious. If he finds “clearly” or “obviously” in either draft, he rewrites it. He verifies every name, date, and causal claim in a trusted source before the material reaches students.
Frank keeps a saved block that ends with the line “Review for any wording that could tilt the reader.” That is his classroom habit, not a finding from the benchmarks. A prompt that suggests an answer is worth checking in any language.
What this means for everyday users
People asking questions in languages other than English use chatbots for ordinary tasks. A parent in Kolkata drafts a note to a teacher. A student in Dhaka checks a homework explanation. A shopkeeper writes a business email. They expect the tool to answer on the facts, not on the framing.
Our findings on multilingual LLM bias give them a practical rule. Read every output twice when the prompt is not in English. Look for any sentence that sounds like agreement with an unstated opinion. Fluent wording can still carry a slant, so mark the phrase that introduces one.
The same rule applies to a document-grounded tool such as NotebookLM. It works from documents you supply, but the teacher still checks its answer against those documents. Do not assume document grounding settles the neutrality question; compare the answer with the supplied text. These results do not establish how a document-grounded tool would respond.
A worked example for a classroom
Say a teacher wants a short explanation of photosynthesis for a mixed-language class. The teacher writes the prompt in English first.
You are helping a middle-school science teacher.
Context: [Students have learned that plants use sunlight, water, and carbon dioxide to make glucose and oxygen.]
Goal: [Draft a short explanation of photosynthesis that students can read in English or Hindi.]
Constraints: [Use plain language. Stay within the facts above. Keep tone neutral and factual.]
Output shape: [Two short paragraphs. Add a glossary of three terms.]
The teacher runs the prompt in English. The output is clean. Then the teacher translates the prompt into Hindi and runs it again. The Hindi output contains a phrase that implies the process is simple and obvious. The teacher removes it.
A bilingual reviewer would mark any phrase that adds a stance the English version avoids. The reviewer would also check whether the explanation stays within the facts and whether the glossary terms match.
Responsible norms that include multilingual LLM bias
The page on responsible AI in education offers a one-page set of norms a class or department can adopt: disclose use, a human decides, verify facts, protect data, fair access. Our research on how models behave differently across languages sits inside the “verify facts” and “fair access” lines. A department could make checking each language it uses part of its verification norm.
The detailed patterns appear in the companion piece on how prompts manipulate models. You can also find related work on the research page.
Common mistakes and quick fixes
A fluent answer does not rule out multilingual LLM bias: compare its wording with a checked version of the prompt. The fix is to run the same prompt in English alongside it and compare specific phrases.
Another mistake is to paste a long prompt that already contains the answer the teacher hopes for. The model then has two signals to follow: the facts and the nudge. The fix is to strip any leading language before translation and to add the explicit instruction below.
Stay strictly neutral. Do not agree or disagree with any implied position.
Teachers sometimes rely on the model to translate the prompt itself. That adds another layer where wording can shift. Before you run a translated prompt, check its meaning with a competent speaker or an appropriate trusted source, especially if you are not fluent in the target language.
A small routine to try
Choose one small task that is due soon. Open a general AI assistant such as ChatGPT, Claude, Gemini or Microsoft Copilot, or a teacher-specific tool such as MagicSchool or Brisk. Write the prompt in English using the seven ingredients: context, goal, constraints, examples, output shape, review, and iteration.
Translate the prompt into Hindi or Bengali, and check the translation with someone who knows the language well. Run both versions. Place the outputs next to each other. Scan for any shift in neutrality. Change the sentences that tilt. Save the prompt with a short note about what you edited.
If no competent speaker can check the translation, use the tool’s own translation as a rough guide, but treat it as a draft. Compare the two outputs for tone and slant. If the second language is one you cannot read, ask a colleague or your school’s language department to review the output before you use it.
Keep student data out of any public tool. Before you paste anything, use the green-yellow-red test. Green means go ahead: generic planning or public information. Yellow means ask your school first, for example with anonymized student work. Red means do not put the material into a tool your school has not approved. Your school’s policy decides the line.
What the study does not claim
The study tested six commercial models. We do not name them publicly, and because we do not name the six models here, these figures cannot guide a choice between named products. They describe patterns across the models we tested.
It covered three languages: English, Hindi and Bengali. It does not claim that every other language will show the same pattern, or that one language is permanently worse.
It measured neutrality under manipulative prompts. A neutral-response rate does not tell you whether an answer is factually correct. It did not test every chatbot task. Maya’s worksheet and Frank’s explanation are illustrations, not tasks from the benchmarks.
Questions teachers ask
Does multilingual LLM bias affect every model the same way?
No. Neutral-response rates ranged from 74.11 percent to 94.54 percent even on the same dataset. The practical step is to test the specific tool you use in your school with the same prompt in two languages.
Can a document-grounded tool remove the risk?
A document-grounded tool works from the documents you provide. These results do not establish how a document-grounded tool would respond. Do not assume that supplying documents settles the neutrality question; check the answer against them.
Should we avoid non-English prompts altogether?
The study does not suggest that. It found increased vulnerability in Hindi and Bengali under manipulative prompts. My practical advice is to check both drafts. Teachers in multilingual classrooms already move between languages. The Draft, Check, Decide method simply travels with them.
How can I check a translation if I am not fluent?
Ask a colleague or your school’s language department to review the output before you use it. If no competent speaker is available, use the tool’s own translation as a rough guide, but treat it as a draft. Compare the two outputs for tone and slant.
The book
The book Keep It Simple: AI for Teachers walks through the same habits in more detail. It explains the five things AI can actually help with: draft, understand, adapt, create, organize. It shows how the seven ingredients turn a vague request into a narrow instruction. The free sample contains the first four chapters.
Open your usual tool tonight. Pick one worksheet or note you need tomorrow. Write the prompt in English, translate it, run both versions, and compare them. If you cannot read the second language, ask a colleague or your school’s language department to check the output before you use it. Change the line that tilts.