At 7:12 p.m. the staff room is quiet except for the hum of the vending machine. Jordan Kim stares at the screen where a general assistant has just produced twenty-seven feedback comments. Some are helpful. Others repeat the same vague praise or contradict notes Jordan wrote last week. The cursor blinks. Before another batch goes live, the model needs checking.
For Jordan, evaluating LLMs starts with checking whether a comment follows the assignment and the student’s paragraph. You need five realistic cases, a narrow task definition, and a habit of logging what breaks.
Start small. Five cases are enough for the first check. Passing them does not establish that the model is reliable.
Define the task narrowly
Write one concrete job the model must do. Vague goals produce vague failures that are hard to spot. Say you want help drafting feedback on student writing about the causes of the American Revolution. The narrow version reads like this.
Draft one sentence of specific praise and one sentence of next-step guidance for each of these [number] student paragraphs on [topic]: [paste student paragraphs]. Use only these [number] historical facts: [list permitted facts]. Match the reading level of a typical [grade level] social studies class. Flag any sentence that could be read two ways. If the assignment prompt is missing, ask for it before drafting.
That single paragraph tells the model what success looks like and what must stay out. It also gives you a clear checklist when you review the output. The prompt is the instruction, not the product. The tool drafts. You check against trusted sources. You decide what reaches students.
A narrow task makes irrelevant or unsupported output easier to identify. A general assistant such as ChatGPT, Claude, Gemini or Microsoft Copilot may still invent details even when the job is bounded. A bounded prompt gives you a clearer standard for spotting invented details; check the output anyway.
A document-grounded tool such as NotebookLM is intended to answer using supplied documents. Check consequential claims against those documents. Teacher-specific tools such as MagicSchool or Brisk follow built-in workflows for common classroom tasks. Tools and their settings change often, so record the documents, prompt, and settings used for every test.
Build a small realistic test set
Collect five examples that match the work you actually do. These become your test set. Cases that resemble the intended work are more useful for testing that task than generic samples you find online.
Before you paste anything, check your school’s policy. Old student paragraphs or teacher notes may need approval. Anonymization alone is not permission. If your school has not approved the tool or the data use, write invented practice cases or use public examples instead.
Keep the set small enough that you can review every output against an answer key. Save the set in a plain document or spreadsheet. You will reuse it every time you try a new model or a major settings change.
Build an answer key for five cases
For each case, write the expected output before you run the test. Include one ordinary case, one ambiguous case, and one case that should trigger a request for missing information.
Say your five cases are student paragraphs on the Stamp Act. For the ordinary and ambiguous cases, you supply the assignment prompt and the permitted facts. For the missing-information case, you deliberately leave the assignment prompt out. Your answer key might look like this.
| Case | Type | Expected result |
|---|---|---|
| 1 | Ordinary | Praise names the specific fact used; guidance points to the missing cause |
| 2 | Ordinary | Praise notes clear topic sentence; guidance suggests adding a date |
| 3 | Ambiguous | Model flags the sentence that could be read two ways instead of guessing |
| 4 | Missing information | Model asks for the assignment prompt before drafting |
| 5 | Ordinary | Praise mentions correct use of evidence; guidance addresses organization |
Case 4 is the one to watch. The student paragraph is included, but the assignment prompt is not, so the model cannot know what the student was asked to argue. The prompt above tells the model to ask for missing assignment context. A careful model asks. A careless one drafts confident feedback anyway.
Here is a short invented paragraph for case 3, the ambiguous one.
The Stamp Act taxed printed materials like newspapers and legal documents, which made many colonists angry enough to act.
The expected result is that the model flags the second half. Act could mean protest, boycott, or violence, and the sentence does not say which. If the model instead guesses one meaning and praises the student for it, that is a failure you record in the log. This is an illustration of what to look for, not a report of what any model did.
When you run the test, compare each output to the key. Note where the model matches, where it misses, and where it adds something you did not ask for.
Evaluating LLMs with rephrasing tests and prompt framing
Run the identical test set through the model with two different phrasings of the same core instruction. Both runs receive the same five cases and the same permitted-facts list. One version might be polite and detailed. Another might be blunt. Then compare the outputs side by side.
Here is a rephrased version of the Stamp Act prompt. The task, facts, reading level, ambiguity rule, missing-information rule, and requested output are identical. Only the wording and tone differ.
You are helping a [grade level] social studies teacher. For each of these [number] student paragraphs on [topic]: [paste student paragraphs], write one sentence that names something the student did well and one sentence that tells them what to fix next. Use only these [number] historical facts: [list permitted facts]. Match the reading level of a typical [grade level] social studies class. If a sentence could be read two ways, flag it. If the assignment prompt is missing, ask for it before drafting.
Same job, different wording. When you compare the two runs, check two things for each case. Did the output flag the ambiguous sentence in case 3? Did the output ask for the missing assignment prompt in case 4? If one phrasing flags ambiguity and the other guesses, that is a material change. If one asks for the prompt and the other drafts anyway, that is a material change.
For case 2, the suggested date must come from the permitted-facts list you supplied. If the model suggests a date that is not on that list, record it as an unsupported factual claim, not a formatting slip.
There is published evidence that framing can push models around. In our paper on how prompts manipulate models, we tested six commercial LLMs with verified translations of established cognitive-bias benchmarks. We analyzed 683,616 model responses across English, Hindi, and Bengali. Confirmation-pattern prompts cut neutral answers by 19.42 percentage points on average. Neutral-response rates ranged from 74.11 percent to 94.54 percent across models on the same dataset. Bias vulnerability increased in Hindi and Bengali relative to English, especially under manipulative prompts.
That study tested manipulative confirmation-pattern prompts, not ordinary polite-versus-blunt rewording. The classroom comparison above is a precaution you can run yourself, not a finding from the paper. You can read more about the research on how prompts manipulate models.
Keep a copy of each prompt next to its output. Use the comparisons to spot differences worth checking on more cases.
A common mistake with evaluating LLMs is treating the first polite prompt as final. Rephrasing may expose a problem; it may not. Say Frank tests two versions of a worksheet prompt. If one drops a learning objective, he logs the difference and checks it again. Record the difference without assuming its cause; repeat the test before drawing a conclusion.
Language tests when students need them
If any of your students work in Hindi, Bengali, or another language, test the model there too. Run the same test set in each language.
Our multilingual bias study found differences on its benchmarks. If families need another language, test the actual messages in that language too. You can read more about the details in our article on multilingual LLM bias.
If you cannot assess the other language yourself, ask a qualified reviewer to check the translated prompt, the expected result, and the model’s output before any material is shared with students or families. Checking only the final output misses errors that entered earlier, in the translation of the task itself.
Here is a concrete procedure for that check. Give the reviewer three things: the original English prompt, the translated prompt, and the model’s output in the target language. Ask the reviewer to confirm that the translated prompt preserves every constraint, including the ambiguity rule and the missing-information rule. Then ask the reviewer to compare the model’s output against the expected result you wrote for that case. If the reviewer finds a shift in tone, a dropped constraint, or an added claim, record it in the failure log.
Denise Carter in the front office runs this test on messages home. One English version reads clearly at a sixth-grade level. The Bengali version adds polite phrasing that softens a required attendance reminder into a suggestion. The failure log catches it before any family sees the note.
Keep a failure log while evaluating LLMs
Create a short table with seven columns: date, task, prompt used, expected result, observed result, severity, and action. Record every clear error, hallucination, or inconsistency. As you add cases, the log may reveal repeat failures.
Here is an example entry.
| Date | Task | Prompt used | Expected result | Observed result | Severity | Action |
|---|---|---|---|---|---|---|
| Oct 3 | Stamp Act feedback | Polite detailed version | Flag ambiguous sentence | Model guessed at meaning | Medium | Rephrase and retest |
| Oct 3 | Stamp Act feedback | Blunt version | Ask for missing assignment prompt | Model drafted anyway | High | Do not use for this task |
The log also reminds you of the limits of any single tool. A document-grounded tool is designed to answer from documents you supply rather than from training alone. But it can still produce unsupported answers, so the teacher must check each consequential claim against those documents. You can read more about how RAG works.
Even these tools can fail when the source documents are incomplete or when the query sits outside the provided text.
Before evaluating LLMs for a science worksheet
Set a stopping rule before you run any cases. Distinguish an immediately disqualifying error from a fixable drafting error. An error that could mislead a student or parent, such as a fabricated fact or a softened safety warning, disqualifies the model for that task. A formatting slip or a repeated phrase is fixable.
Use only tools and data your organization’s or school’s policy permits. If the policy blocks the tool or the data, stop there. Do not test with real student work unless the school has approved both the tool and the data use.
Picture Maya Rodriguez testing a model on energy transfer. She defines the task with seven prompt ingredients. Context names the grade, the objective, and the exact facts students have learned. Goal limits the output to a 15-minute worksheet. Constraints ban labs, homework, and invented standards. One example question shows the voice she wants. Output shape separates the student page from the answer key. Review tells the model to flag every scientific claim. Iteration saves the prompt for next week.
She builds five test cases. One ordinary set of facts. One with a common misconception about cold moving from a mug to a spoon. One missing the exact objective. Two more that vary only in reading level.
Maya runs the first prompt. The model produces eight questions. Three of them repeat the same idea. The answer key claims thermal energy disappears. She marks those in the failure log as high-severity because they directly contradict the facts she supplied.
Next she rephrases the prompt. The second version yields a cleaner structure but invents a new standard. The log now shows two different failures on identical facts. That pattern tells her the model is not stable enough for this job without heavy checking.
She translates the prompt into Bengali for a student who needs it. The output adds extra polite language that changes the tone of the misconception note. A colleague confirms the shift. The log gains another row.
After ten minutes of review Maya closes the file. The model fails three out of five cases on the first pass and two on the rephrased pass. Checking and fixing takes longer than writing the worksheet herself. She walks away and tries a different general assistant the next evening.
The routine took twenty-two minutes total. The failure log now holds three entries she can reuse when she tests the next model. These times and counts are fictional details, not typical evaluation speeds or results.
Evaluating LLMs outside the classroom: a worked example
Say Jordan needs to draft a project status update from a short brief. The permitted source is the brief itself. No outside facts. The task is to produce three sentences: what was done, what is blocked, what happens next.
Jordan writes five cases. Case 1 is a brief with clear progress and one blocker. Case 2 has no blocker listed. Case 3 has a blocker but no next step. Case 4 is missing the date. Case 5 is a brief with two conflicting progress claims.
The expected results are written before any test. Case 1 should produce three sentences that match the brief. Case 2 should state that no blocker was reported. Case 3 should ask for the next step. Case 4 should ask for the date. Case 5 should flag the conflict instead of picking one claim.
Jordan runs the first phrasing. The model handles cases 1 and 2. On case 3 it invents a next step. On case 4 it drafts without the date. On case 5 it picks one claim and ignores the other.
The second phrasing uses blunter wording. The model now asks for the date on case 4 but still invents a next step on case 3. The log shows the same failure under both phrasings.
When the output is consequential, a recommended safeguard is to have someone besides the person who wrote the prompt review it, subject to the organization’s policy. Jordan’s status update goes to a project lead who did not run the test. The log lets the lead compare the failure with the stopping rule set before testing. If an output cannot be adequately checked, do not use it. Five cases do not validate a production system. They show where the model broke on this task, with these prompts, under these settings.
Common mistakes when evaluating LLMs
A common mistake with evaluating LLMs is skipping the answer key. Without it you compare the output to your memory instead of to a fixed standard. Fix it by writing the key first, before you ever paste the test set.
Another mistake is using the same phrasing every time. The model may appear consistent only because you never varied the frame. Run the rephrasing test on every new task.
Some teachers test only in English even when families speak other languages at home. Our multilingual bias study found differences on its benchmarks. If families need another language, test the actual messages in that language too.
Many stop at the first failure. One error does not doom a model. The log lets you compare the failure with the stopping rule you set before testing.
Finally, some treat the model output as final instead of a draft. Checking and deciding remain the teacher’s responsibility, subject to school policy.
Questions teachers ask
How many cases do I really need when evaluating LLMs?
Five is enough to start. More cases can expose more failure types, but a small test set cannot establish reliability for a high-stakes use. The goal is coverage of ordinary, ambiguous, and missing-information cases, not statistical significance. Add cases only after you have logged the first round.
Can I use this routine with a teacher-specific tool?
Yes. The steps stay the same. A teacher-specific tool may already embed some constraints, yet it can still hallucinate or ignore your exact facts. Test it the same way you would test a general assistant.
What if my school only allows one approved model?
Evaluate that model anyway. The log can show failures you have found and tasks that still require close review. You still decide what reaches students.
How do I know when the failure log says walk away?
Look at severity and time. Set a stopping rule for the particular task. If the model produces errors you cannot safely catch or correct, stop immediately. If you spend longer fixing the draft than creating it yourself, the model is not saving time. Try another tool.
Does this work for document-grounded tools?
It does. Feed the tool the same five test documents and the same questions. Check whether every factual sentence points back to a source you supplied. The RAG explained article shows where these tools still fail even when they cite a document.
How long should I keep the failure log?
Keep it until you settle on a stable set of tools. After that, add one row each time you test a major update or a new task type.
When to walk away
Review the failure log after every new model or major update. Set a stopping rule based on error severity and the time you spend checking and repairing outputs. If the model produces errors that could mislead students or parents, or if you spend more time fixing than drafting, it is not saving time. Walk away and try a different general assistant, a document-grounded tool, or one of the teacher-specific tools built for common classroom tasks.
Some failures cannot be patched with better prompts. When serious errors recur in the test, stop relying on rephrasing. Move to another tool and keep the log for the next comparison.
The checking habit behind this routine is the same one that runs through Keep It Simple: AI for Teachers. The free sample covers the opening chapters if you want to see the approach.
For a broader framework on managing AI risk, see the NIST AI Risk Management Framework.
Pick one low-risk task your school’s policy already allows, the kind you could hand to a general assistant without student data attached. Write one test case and its expected result in a blank document. Run it through a model you already have access to. Record what the model does in the first row of your failure log.