Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

IMPORTANCE The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians.OBJECTIVE To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions.DESIGN, SETTING, AND PARTICIPANTS In this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit's r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose "which response was better" and judged both "the quality of information provided" (very poor, poor, acceptable, good, or very good) and "the empathy or bedside manner provided" (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians.RESULTS Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6%(95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P

重要的是，虚拟医疗保健的迅速扩展导致患者信息的激增与医疗保健专业人员之间的更多工作和倦怠有关。人工智能（AI）助理可以通过起草可以审查的临床医生来评估AI Chatbot Assistant（ChatGPT）的能力，可能会通过起草可以审查的回答来帮助为患者问题提供答案在此横断面研究中进行设计，设置和参与者，这是一个公共社交媒体论坛中的公共和不可识别的问题的数据库（Reddit的R/AskDocs）用于从2022年10月开始随机进行195次交流，一位经过验证的医生回答了公共问题。聊天机器人的回答是通过将原始问题输入到2022年12月22日至23日的新的会话中（在会议上没有提出的事先问题）来产生。由有执照的医疗保健专业人员组成的团队。评估人员选择了“哪种响应更好”，并判断“提供的信息质量”（非常贫穷，贫穷，可接受，良好或非常好）和“所提供的同理心或床边方式”（不是同理心，略有同情心，中等同情心，善解人意，非常善解人意）。平均结果以1至5级订购，并在聊天机器人和医师之间进行比较。在195个问题和回答中，评估者在78.6％的聊天机器人的反应中更喜欢聊天机器人的反应（95％CI，75.0％，75.0％-81.8％） 。平均（IQR）医师的反应明显短于聊天机器人反应（52 [17-62]单词vs 211 [168-245]单词； t = 25.4; p