Measuring the Impact of Explanation Bias: A Study of Natural Language Justifications for Recommender Systems

Despite the potential impact of explanations on decision making, there is a lack of research on quantifying their effect on users’ choices. This paper presents an experimental protocol for measuring the degree to which positively or negatively biased explanations can lead to users choosing suboptimal recommendations. Key elements of this protocol include a preference elicitation stage to allow for personalizing recommendations, manual identification and extraction of item aspects from reviews, and a controlled method for introducing bias through the combination of both positive and negative aspects. We study explanations in two different textual formats: as a list of item aspects and as fluent natural language text. Through a user study with 129 participants, we demonstrate that explanations can significantly affect users’ selections and that these findings generalize across explanation formats.

尽管解释对决策可能存在影响，但在量化其对用户选择的作用方面却缺乏研究。本文提出了一种实验方案，用于衡量正向或负向偏差的解释在多大程度上会导致用户选择次优推荐。该方案的关键要素包括一个偏好诱导阶段（以便实现推荐的个性化）、从评论中手动识别和提取物品的各个方面，以及一种通过正负两方面相结合来引入偏差的可控方法。我们研究了两种不同文本格式的解释：作为物品方面的列表以及作为流畅的自然语言文本。通过对129名参与者进行的用户研究，我们证明了解释会显著影响用户的选择，并且这些发现适用于不同的解释格式。