Validity and Reliability of Automatically Generated Propositional Reasoning Items A Multilingual Study of the Challenges of Verbal Item Generation

This study introduces a newly developed public-domain multilingual automatic item generator that creates propositional reasoning (PR) items belonging to 15 item families by using various inference rules. Psychometric properties of the resulting written PR test were investigated in three diverse samples in English, simplified Chinese, and German, respectively. Internal consistency was good to excellent across samples. The ICAR16 short form test of cognitive abilities (Condon & Revelle, 2014) was used to evaluate construct validity. Correlations of ICAR16 scores and PR scores were high. Furthermore, items within families appeared to be equivalent, with only minor differential item functioning between the Chinese- and English-speaking samples. Performance on the PR test was shown to be reasonably stable over the course of 1 week. Differences of total scores between test forms (pen and paper vs. computerized administration) were not detected. Findings suggest that the automatically generated PR test is a valuable instrument for the assessment of propositional reasoning ability.

本研究介绍了一种新开发的公共领域多语言自动题目生成器，它通过使用各种推理规则来创建属于15个题目家族的命题推理（PR）题目。分别在英语、简体中文和德语的三个不同样本中对由此产生的书面命题推理测试的心理测量特性进行了研究。各样本的内部一致性良好至优秀。使用了认知能力的ICAR16简式测试（康登和雷维尔，2014）来评估结构效度。ICAR16分数和命题推理分数的相关性很高。此外，题目家族内部的题目似乎是等效的，在说中文和说英语的样本之间只有较小的题目功能差异。命题推理测试的成绩在1周的时间内表现出相当的稳定性。未检测到测试形式（纸笔测试与计算机化测试）之间总分的差异。研究结果表明，自动生成的命题推理测试是评估命题推理能力的一种有价值的工具。