Addressing Algorithmic Unreliability and Dataset Shift in EHR-based Risk Prediction Models

解决基于 EHR 的风险预测模型中的算法不可靠性和数据集转移

• 批准号: 10679376
• 负责人: Likhitha Kolla
• 金额: $ 4.77万
• 依托单位: UNIVERSITY OF PENNSYLVANIA
• 依托单位国家: 美国
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-06-01 至 2026-05-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10679376
• 关键词: Accident and Emergency department; Address; Advanced Malignant Neoplasm; Algorithms; Biometry; COVID-19; COVID-19 pandemic; Caring; Case Management; Cessation of life; Characteristics; Clinical; Code; Collection; Compensation; Data; Data Set; Data Sources; Deterioration; Development; Development Plans; Early Intervention; Electronic Health Record; Environment; Future; Generations; Guidelines; Health; Health system; Healthcare; Hospital Costs; Hospitalization; Impairment; Incentives; Inpatients; Institution; Interruption; Intervention; Knowledge; Laboratories; Machine Learning; Maintenance; Methods; Mission; Modeling; Needs Assessment; Onset of illness; Outcome; Outpatients; Patient Care; Patients; Performance; Physicians; Play; Policies; Policy Maker; Population; Predictive Analytics; Primary Care; Process; Public Health; Qualifying; Quarantine; Refit; Reporting; Research; Research Personnel; Resource Allocation; Resources; Risk Adjustment; Role; Scientist; Sepsis; Series; System; Testing; Time; Training; Triage; United States National Institutes of Health; Validation; Veterans Health Administration; Work; analytical tool; beneficiary; care delivery; career development; clinical application; clinical decision-making; clinical predictive model; demographics; design; doctoral student; evidence base; health care disparity; health care service utilization; health care settings; health service use; high risk; improved; innovation; model design; model development; mortality risk; operation; outpatient programs; pandemic disease; pandemic impact; prediction algorithm; predictive modeling; predictive tools; prevent; provider behavior; research and development; risk prediction; risk prediction model; tool

Project Summary Predictive analytic algorithms built on electronic health record (EHR) inputs, such as patient characteristics, administrative codes, and lab values, are increasingly used in health care settings to direct resources to high- risk patients. Data play an indispensable role in the development and deployment of effective predictive models. The greatest, yet understudied, challenge in the maintenance of these tools arises from a data-related concern, namely dataset shift, in which training data distribution differs from the population on which the algorithm is deployed, leading to model deterioration and inaccurate risk predictions. Dataset shift is a pervasive cause of algorithmic unreliability in EHR-based models due to inevitable changes in physician behaviors and health system operations that alter (1) the input distribution (covariate drift); and (2) changes in the relationship between predictors and outcome (concept drift). Sudden changes in healthcare utilization during the COVID-19 pandemic may have impacted the data generation process and the performance of clinical predictive models. Our preliminary study showed that decreased collection of patient labs during the COVID-19 quarantine period led to sparse data generation for important predictors of a single-institution EHR-based mortality risk prediction algorithm, underpredicting risk for patients with advanced cancers. Despite the increasing use of predictive tools in high stakes clinical applications; and growing recognition of dataset shift, we lack a framework for reasoning shift and its effects on care delivery; and for proactively addressing shift to maintain performance over time. In Aim 1, we propose to extend prior works on shift to a nationally deployed risk prediction algorithm, the VA Care Assessment Need (CAN) model, used on millions of VA beneficiaries each year. The VA CAN model predicts the likelihood of hospitalization within 90 days or 1 year after a primary care encounter to identify high-risk patients who would benefit from additional outpatient interventions. We also investigate covariate and concept drift as two possible mechanisms for COVID-19 associated dataset shift. In Aim 2, we apply an interrupted time series design to study the association between sudden shift at the onset of the pandemic on case-management decisions. Current solutions to address dataset shift have primarily been reactive (i.e. model retraining with recent data), however, fail to be robust in new testing environments. In Aim 3, we consider revision of the VA CAN model via machine learning and inclusion of variables that reflect potential drivers of shift. This project is innovative as it is the first to leverage a rigorous statistical framework to study extent and mechanisms of shift and develop proactive guidelines for model maintenance. The training plan is rigorous for Ms. Kolla, an MD-PhD student in biostatistics. She is strongly supported by her department and institution as well as her two high- qualified sponsors: Dr. Jinbo Chen, an expert in EHR-based risk prediction modeling, and Dr. Ravi Parikh, an expert in implementation of predictive analytics. The proposed research and career development plan will be an essential step towards Ms. Kolla’s development as an interdisciplinary and independent physician-scientist.

项目摘要 基于电子健康记录（EHR）输入的预测分析算法，例如患者特征， 行政代码和实验室价值越来越多地用于医疗保健环境中，以将资源引导到高级 风险患者。数据在有效预测模型的开发和部署中起着必不可少的作用。 在维护这些工具方面，最大，但最受理解的挑战是由数据相关的关注所引起的， 即数据集的转移，其中培训数据分布与算法所在的人群的差异 部署，导致模型定义和不准确的风险预测。数据集偏移是一个普遍的原因 由于身体行为和健康的不可避免的变化，基于EHR模型的算法不可靠性 更改（1）输入分布（协变性漂移）的系统操作； （2）改变 预测因素和结果（概念漂移）。 19009年大流行期间医疗保健利用的突然变化 可能已经影响了数据生成过程和临床预测模型的性能。我们的 初步研究表明，在COVID-19隔离期LED期间，患者实验室的收集减少 为基于EHR EHR的重要预测指标的重要预测指标稀疏数据生成 算法，对晚期癌症患者的预测风险不足。尽管使用预测工具越来越多 在高股份临床应用中；以及对数据集转移的认识日益认识，我们缺乏推理的框架 转移及其对护理提供的影响；并主动解决转移以保持绩效随着时间的流逝。在 AIM 1，我们建议将改动的先前工作扩展到全国部署的风险预测算法，即VA护理 评估需求（CAN）模型，每年用于数百万VA受益人。 VA可以模拟预测 初级保健遇到高危高危后90天或1年内住院的可能性 将受益于其他门诊干预措施的患者。我们还研究协变量和概念 作为Covid-19相关数据集偏移的两个可能的机制漂移。在AIM 2中，我们应用了一个中断的时间 串联设计以研究大流行病开始时突然转移之间的关联 决定。当前解决数据集偏移的解决方案主要反应性（即使用 但是，最近的数据）在新的测试环境中无法强大。在AIM 3中，我们考虑对VA的修订 可以通过机器学习和包含变量来建模，这些变量反映了潜在的偏移驱动因素。这个项目是 创新性，因为它是第一个利用严格的统计框架来研究范围和转移机制 并制定积极的模型维护准则。 MD-PHD Kolla女士对培训计划非常严格 生物统计学的学生。她的部门和机构以及她的两个高级人士都得到了强烈的支持 合格的赞助商：基于EHR的风险预测建模专家Jinbo Chen博士和Ravi Parikh博士 实施预测分析的专家。拟议的研究和职业发展计划将是 作为跨学科和独立的身体科学家发展的基本步骤。