How Bias Cues Affect Trust in Large Language Model Responses: An Empirical Study Design and Evaluation
2026 (English)Independent thesis Basic level (degree of Bachelor), 10 credits / 15 HE credits
Student thesisAlternative title
Hur biassignaler påverkar tilliten till svar från stora språkmodeller : En empirisk studiedesign och utvärdering (Swedish)
Abstract [en]
Large language models (LLMs) are increasingly integrated into critical decision-making processes, serving as assistants for education, writing, and professional support. This widespread adoption places questions of bias and trust at the center of their responsible deployment. However, much of the existing research on LLM bias focuses on evaluating the internal mechanics and probabilities of models, rather than exploring how users perceive and interact with generated text in real-world scenarios. The core research problem addressed in this thesis is the misalignment between true model capabilities and human perception, often termed the calibration gap. Prior research establishes that state-of-the-art LLMs exhibit persistent social identity and cognitive biases despite rigorous value alignment [1, 2, 3]. Furthermore, users often systematically overestimate LLM accuracy when presented with confident or detailed explanations [4]. Parallel work demonstrates that superficial presentation cues, such as attribution or subtle framing, can systematically shift how information is evaluated [5]. Building upon this theoretical foundation, this thesis investigates how observable bias cues embedded in LLM responses, specifically operationalized as subtle positive or negative framing relative to a neutral baseline, influence readers’ perceived trustworthiness and related judgments. To examine this, we conducted an online survey experiment (N = 42) wherein participants evaluated five realistic AI assistant scenarios. For each scenario, participants were randomized to read one of three response variants and subsequently rated the output on four multidimensional Likert items: trustworthiness, objectivity, decision comfort, and appropriateness of tone. Our quantitative and qualitative analysis reveals that framing cues primarily affected perceptions of objectivity and tone, with both positively and negatively framed variants being rated as less objective and less appropriate than their neutral counterparts. Interestingly, the direct effects on overall factual trustworthiness were highlyscenario-dependent, yielding statistically significant differences in only two out of the five scenarios. Additionally, we found that a user’s baseline trust in AI correlated strongly and positively with their trustworthiness ratings across all scenarios(r = 0.51, p = 0.001). Taken together, the findings suggest that users often decouple factual trustworthiness from perceived appropriateness. Even when an LLM employs promotional or restrictive language, users may still trust the underlying information, particularly if their baseline confidence in AI is high. These results emphasize that trust is a complex, multidimensional construct influenced as much by stable individual predispositions as by the generated text itself. The study contributes to the ongoing discourse on LLM trust calibration by providing empirical, user-centered evidence on cue sensitivity, highlighting the necessity of designing systems that prioritize neutral presentation to mitigate the insidious effects of framing biases.
Place, publisher, year, edition, pages
2026. , p. 17
Keywords [en]
large language models, bias cues, trust, calibration, survey experiment, cog- nitive bias, framing
National Category
Other Engineering and Technologies
Identifiers
URN: urn:nbn:se:lnu:diva-149359OAI: oai:DiVA.org:lnu-149359DiVA, id: diva2:2097167
Subject / course
Computer Science
Educational program
Software Technology Programme, 180 credits
Presentation
2026-08-10, D3373V, Vejdes plats 7, Växjö, 09:00 (English)
Supervisors
Examiners
2026-09-012026-08-312026-09-01Bibliographically approved