The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?

ChatGPT, a language-learning model chatbot, has garnered considerable attention for its ability to respond to users’ questions. Using data from 14 countries and 186 institutions, we compare ChatGPT and student performance for 28,085 questions from accounting assessments and textbook test banks. As of January 2023, ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. We provide evidence of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. We also discuss implications for accounting education and research.

ChatGPT是一种语言学习模型聊天机器人，因其能够回答用户问题的能力而受到了相当多的关注。我们利用来自14个国家和186个机构的数据，针对会计评估和教材测试题库中的28085个问题，对ChatGPT和学生的表现进行了比较。截至2023年1月，ChatGPT对56.5%的问题提供了正确答案，对另外9.4%的问题提供了部分正确的答案。当考虑问题的分值时，如果不给予部分分数，学生在评估中的平均成绩（76.7%）显著优于ChatGPT（47.5%）；如果给予部分分数，ChatGPT的成绩为56.5%。然而，当我们考虑部分分数时，ChatGPT在15.8%的评估中表现优于学生的平均水平。我们提供了ChatGPT在不同问题类型、会计主题、课程级别、开放式/封闭式评估以及测试题库问题上的表现证据。我们还讨论了其对会计教育和研究的影响。