Evaluating Prompt Variations in AI-Generated Grammar ExplanationsAn Information Quality Approach
2026 (English) Independent thesis Advanced level (degree of Master (One Year)), 10 credits / 15 HE credits
Student thesis
Abstract [en]
The increasing use of large language models (LLMs) for educational content generation highlights the need for systematic evaluation of AI-generated materials. Educational content generation raises concerns about output quality, including accuracy, completeness, clarity, and relevance, and research indicates that these qualities can vary depending on how prompts are formulated. This study conducts a controlled comparative evaluation of four predefined prompt formulations — Minimal Instruction, Structured Instruction, Pedagogically Anchored Instruction, and Length-Constrained Instruction — and their effect on the Information Quality (IQ) of CEFR-aligned grammar explanations in English. Using a within-subjects experimental design, the same 20 predefined grammar tasks were generated under each prompt condition within a single LLM (GPT-5.3) environment, maintaining constant model parameters and evaluation procedures throughout. Generated explanations are evaluated as information artifacts using four IQ dimensions: Accuracy, Completeness, Clarity, and Relevance. Evaluation is conducted blind to prompt condition using a standardised 5-point Likert scale rubric with explicit scoring anchors, applied independently by ten practising English teachers.
The dominant finding is one of uniformly high output quality across all prompt conditions. No statistically significant differences were detected across any of the four IQ dimensions — Accuracy (p = .515), Completeness (p = .296), Clarity (p = .746), or Relevance (p = .358) — and effect sizes were negligible throughout. Inter-rater reliability values near zero reflect a ceiling effect in the score distribution rather than rater disagreement, and are consistent with the finding of uniformly high output quality. These results suggest that advanced LLMs have become sufficiently robust to produce high-quality grammar explanations from minimal prompts, with structured prompt engineering functioning as a fine-tuning mechanism rather than a primary quality driver. The study contributes a replicable evaluation procedure for AI-generated educational content and provides empirical grounding for reconsidering the role of prompt complexity in LLM-assisted language instruction.
Place, publisher, year, edition, pages 2026. , p. 53
Keywords [en]
Prompt engineering, prompt variation, large language models (LLMs), generative AI, information quality, AI-generated content evaluation, educational content, grammar explanation
National Category
Natural Sciences Natural Language Processing
Identifiers URN: urn:nbn:se:lnu:diva-149661 OAI: oai:DiVA.org:lnu-149661 DiVA, id: diva2:2102420
Subject / course Informatics
Educational program Master Programme in Information Systems, 60 credits
Supervisors
Examiners
2026-09-212026-09-172026-09-21 Bibliographically approved