PREMIERE: A PREdictive Model Index and Exchange REpository

PREMIERE：预测模型索引和交换存储库

• 批准号: 10016297
• 负责人: ALEX BUI
• 金额: $ 67.35万
• 依托单位: UNIVERSITY OF CALIFORNIA LOS ANGELES
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-15 至 2024-05-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10016297
• 关键词: Access to Information; Address; Algorithms; Area; Attention; Bayesian Network; Big Data; Biological Markers; Calibration; Characteristics; Clinical; Communities; Computational Biology; Computer software; Data; Data Science; Data Set; Data Sources; Decision Making; Decision Trees; Dermatology; Development; Diagnosis; Diagnostic; Diagnostic Imaging; Ecosystem; Educational workshop; Electronic Health Record; Environment; Evaluation; FAIR principles; Fostering; Foundations; Goals; Human; Image; Image Analysis; Informatics; Language; Link; Literature; Machine Learning; Measures; Medical; Metadata; Methodology; Methods; Modeling; Nature; Ophthalmology; Pathway interactions; Performance; Population; Publications; Publishing; Radiology Specialty; Receiver Operating Characteristics; Reporting; Reproducibility; Reproduction; Research Personnel; Risk Assessment; Source; Techniques; Testing; Training; Validation; Variant; Work; base; bioimaging; biomarker discovery; case-based; cohort; collaborative environment; comparative; computer aided detection; convolutional neural network; data sharing; deep learning; design; experience; feature selection; indexing; innovation; insight; interest; interoperability; learning network; lung basal segment; lung cancer screening; mHealth; machine learning method; model development; novel; novel strategies; online repository; predictive modeling; prognostic; repository; software repository; statistical and machine learning; statistics; stem; tool; web portal

The confluence of new machine learning (ML) data-driven approaches; increased computational power; and access to the wealth of electronic health records (EHRs) and other emergent types of data (e.g., omics, imaging, mHealth) are accelerating the development of biomedical predictive models. Such models range from traditional statistical approaches (e.g., regression) through to more advanced deep learning techniques (e.g., convolutional neural networks, CNNs), and span different tasks (e.g., biomarker/pathway discovery, diagnostic, prognostic). Two issues have become evident: 1) as there are no comprehensive standards to support the dissemination of these models, scientific reproducibility is problematic, given challenges in interpretation and implementation; and 2) as new models are put forth, methods to assess differences in performance, as well as insights into external validity (i.e., transportability), are necessary. Tools moving beyond the sharing of data and model “executables” are needed, capturing the (meta)data necessary to fully reproduce a model and its evaluation. The objective of this R01 is the development of an informatics standard supporting the requisite information for scientific reproducibility for statistical and ML-based biomedical predictive models; from this foundation, we then develop new computational approaches to compare models' performance. We begin by extending the current Predictive Model Markup Language (PMML) standard to fully characterize biomedical datasets and harmonize variable definitions; to elucidate the algorithms involved in model creation (e.g., data preprocessing, parameter estimation); and to explain the validation methodology. Importantly, models in this PMML format will become findable, accessible, interoperable, and reusable (i.e., following FAIR principles). We then propose novel meth- ods to compare and contrast predictive models, assessing transportability across datasets. While metrics exist for comparing models (e.g., c-statistics, calibration), often the required case-level information is not available to calculate these measures. We thus introduce an approach to simulate cases based on a model's reported da- taset statistics, enabling such calculations. Different levels of transportability are then assigned to the metrics, determining the extent to which a selected model is applicable to a given population/cohort (i.e., helping answer the question, can I use this published model with my own data?). We tie these efforts together in our proposed framework, the PREdictive Model Index & Exchange REpository (PREMIERE). We will develop an online portal and repository for model sharing around PREMIERE, and our efforts will include fostering a community of users to guide its development through workshops, model-thons, and other activities. To demonstrate these efforts, we will bootstrap PREMIERE with predictive models from a targeted domain (risk assessment in imaging-based lung cancer screening). Our efforts to evaluate these developments will engage a range of stakeholders (model developers, users) to inform the completeness of our standard; and biostatisticians and clinical experts to guide assessment of model transportability.

新机器学习的汇合（ML）数据驱动的方法； 获取电子健康记录（EHR）和其他新兴数据类型的财富（例如，OMICS，Imaging，Imaging， MHealth）正在加速生物医学预测模型的发展。 统计方法（例如，回归）到更先进的深度学习技术（例如卷积 神经网络，CNN）和跨越不同的任务（例如，生物标志物/途径发现，诊断，程序组织）。 两个问题变得显而易见：1）因为没有支持传播的全面标准 在解释和实施方面，这些模型，科学的繁殖是有问题的 2）随着新模型的提出，评估性能差异的方法以及对外部的见解 有效性（即可运输能力）是领口。 需要，捕获完全繁殖模型并进行评估所需的（元）数据。 R01的目的是开发信息标准，以支持必要的信息形成？ 基于统计和ML的生物医学预测模型的科学繁殖； 开发新的委员会方法来比较模型的性能。 预测模型标记语言（PMML）标准，以完全表征生物医学数据集并协调 可变定义； 估计）；为了解释验证方法。 可以找到，可访问，可互操作和可重复使用（即，遵循公平的原则）。 比较和对比预测模型的OD，评估跨数据集的可运输性。 为了比较模型（例如，C统计数据，校准），通常无法使用所需的案例级信息 计算这些措施。 taset统计，实现这种计算。 确定所选模型适用于给定的人群/队列的程度（即帮助答案 问题，我可以使用自己的数据使用已发布的模型吗？） 框架，预测模型索引和交换存储库（首映）。 以及用于首映围绕模型共享的存储库，我们的敌方将包括培养用户的评论 通过研讨会，模特和其他活动来指导其发展。 我们将以来自目标域的预测模型引导首映 肺癌筛查）。 开发人员，用户）为我们的标准和临床专家提供了信息 评估模型可运输性。