Learning Universal Patient Representations with Hierarchical Transformers

使用分层转换器学习通用患者表示

• 批准号: 10587270
• 负责人: Timothy A Miller
• 金额: $ 64.82万
• 依托单位: BOSTON CHILDREN'S HOSPITAL
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-02-07 至 2027-07-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10587270
• 关键词: Accident and Emergency department; Address; Algorithms; Architecture; Area; Chest; Classification; Clinical; Clinical Research; Code; Competence; Complex; Computers; Data; Data Aggregation; Data Set; Databases; Decision Making; Development; Diagnosis; E-learning; Electronic Health Record; Experimental Designs; Funding; Goals; Health Status; Image; Individual; Injury; Intensive Care Units; Knowledge; Learning; Length; Link; Machine Learning; Maps; Medical; Methods; Modality; Modeling; Output; Patient Care; Patient Education; Patients; Performance; Phenotype; Privatization; Process; Productivity; Research; Severities; Source; Structure; Supervision; System; Techniques; Terminology; Text; Time; Training; Unified Medical Language System; Work; data fusion; experimental study; frontier; health data; improved; innovation; language training; light weight; machine learning algorithm; multimodality; multiple data types; neural; neural model; novel; patient prognosis; point of care; prevent; prognostic; structured data; usability

Project Summary Electronic health records contain a wealth of information about patient health status that can be mined for multiple purposes, including clinical research and improved decision-making at the point of care. This information can be represented as structured variables, unstructured text, and images, among other data types. In this work, we develop new models for representing the unstructured text that take advantage of powerful neural models called pre-trained transformers. We propose to make these models usable for much longer texts by adding hierarchical layers to operate over summaries of smaller chunks of text, and shrinking the size of the encoder that operates on smaller chunks. First, we develop a smaller encoder for sentence and paragraph-sized texts, by using a technique called extreme distillation that trains smaller models from the output of larger models. We also propose to pre-train hierarchical models for text, by taking advantage of smaller encoders like that from the first aim. We take advantage of both public and private datasets and experiment with different pre-training tasks and architectures. Our final aim proposes to combine representations learned from text with those from the more mature areas of structured data and images. We design experiments that answer the question of how best to merge these different information sources, and apply them to important clinical classification use cases that are likely to require multiple information sources for accurate performance. Specifically, we address the clinical tasks of predicting injury severity in emergency departments, and predicting diagnosis and prognosis of patients in intensive care units.

项目摘要 电子健康记录包含有关患者健康状况的大量信息 多种目的，包括临床研究和改善护理时的决策。这 信息可以表示为结构化变量，非结构化文本和图像，以及其他数据 类型。在这项工作中，我们开发了代表非结构化文本的新模型 强大的神经模型称为预训练的变压器。我们建议使这些模型可用于很多 较长的文本通过添加层次层来对较小的文本的摘要进行操作，然后收缩 在较小的块上运行的编码器的大小。首先，我们为句子开发了一个较小的编码器， 段落大小的文本，使用一种称为“极端蒸馏”的技术，该技术从训练较小的模型中 较大型号的输出。我们还建议通过利用 较小的编码器从第一个目标出发。我们利用公共和私人数据集以及 尝试不同的训练前任务和体系结构。我们的最终目标提议结合 从文本中汲取的表示，从结构化数据和图像的更成熟领域的文本中学到。我们 设计实验回答了如何最好地合并这些不同信息源的问题 将它们应用于可能需要多个信息源的重要临床分类用例 为了准确的性能。具体而言，我们解决了预测紧急情况伤害严重程度的临床任务 部门，预测重症监护病房中患者的诊断和预后。