lnu.sePublications
System disruptions
There are recurring service disruptions on some of the search portals due to high traffic. We are working to resolve the issue, you may temporarily encounter an error message.
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Evaluating Prompt Variations in AI-Generated Grammar ExplanationsAn Information Quality Approach
Linnaeus University, Faculty of Technology, Department of Informatics.
Linnaeus University, Faculty of Technology, Department of Informatics.
2026 (English)Independent thesis Advanced level (degree of Master (One Year)), 10 credits / 15 HE creditsStudent thesis
Abstract [en]

The increasing use of large language models (LLMs) for educational content generation highlights the need for systematic evaluation of AI-generated materials. Educational content generation raises concerns about output quality, including accuracy, completeness, clarity, and relevance, and research indicates that these qualities can vary depending on how prompts are formulated. This study conducts a controlled comparative evaluation of four predefined prompt formulations — Minimal Instruction, Structured Instruction, Pedagogically Anchored Instruction, and Length-Constrained Instruction — and their effect on the Information Quality (IQ) of CEFR-aligned grammar explanations in English. Using a within-subjects experimental design, the same 20 predefined grammar tasks were generated under each prompt condition within a single LLM (GPT-5.3) environment, maintaining constant model parameters and evaluation procedures throughout. Generated explanations are evaluated as information artifacts using four IQ dimensions: Accuracy, Completeness, Clarity, and Relevance. Evaluation is conducted blind to prompt condition using a standardised 5-point Likert scale rubric with explicit scoring anchors, applied independently by ten practising English teachers.

The dominant finding is one of uniformly high output quality across all prompt conditions. No statistically significant differences were detected across any of the four IQ dimensions — Accuracy (p = .515), Completeness (p = .296), Clarity (p = .746), or Relevance (p = .358) — and effect sizes were negligible throughout. Inter-rater reliability values near zero reflect a ceiling effect in the score distribution rather than rater disagreement, and are consistent with the finding of uniformly high output quality. These results suggest that advanced LLMs have become sufficiently robust to produce high-quality grammar explanations from minimal prompts, with structured prompt engineering functioning as a fine-tuning mechanism rather than a primary quality driver. The study contributes a replicable evaluation procedure for AI-generated educational content and provides empirical grounding for reconsidering the role of prompt complexity in LLM-assisted language instruction.

Place, publisher, year, edition, pages
2026. , p. 53
Keywords [en]
Prompt engineering, prompt variation, large language models (LLMs), generative AI, information quality, AI-generated content evaluation, educational content, grammar explanation
National Category
Natural Sciences Natural Language Processing
Identifiers
URN: urn:nbn:se:lnu:diva-149661OAI: oai:DiVA.org:lnu-149661DiVA, id: diva2:2102420
Subject / course
Informatics
Educational program
Master Programme in Information Systems, 60 credits
Supervisors
Examiners
Available from: 2026-09-21 Created: 2026-09-17 Last updated: 2026-09-21Bibliographically approved

Open Access in DiVA

fulltext(1864 kB)21 downloads
File information
File name FULLTEXT01.pdfFile size 1864 kBChecksum SHA-512
915220fd610ea52b8500858ec459e21f1928ee8c577bdc2158b1e1f85064251d891b415f71aa35291f5ce6b5e607b744233aec8b36a3d0446d799682ed4de584
Type fulltextMimetype application/pdf

By organisation
Department of Informatics
Natural SciencesNatural Language Processing

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 130 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf