Cross-level Convolutional Transformer and Adversarial Multi-task Learning for Medical Semantic Segmentation

用于医学语义分割的跨级卷积变压器和对抗性多任务学习

• 批准号: 2722537
• 金额: --
• 依托单位: University College London
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2022
• 资助国家: 英国
• 起止时间: 2022 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2722537
• 关键词: Cross; level; Convolutional; Transformer; Adversarial

This project plans to study cross-level convolutional Transformers and adversarial multi-task learning for medical semantic segmentation (MSS). The goal of MSS labels each pixel of an image with a corresponding class of what is being represented to provide the segmentation maps. Though deep learning-based methods have achieved state-of-the-art performance in MSS, they still struggle to achieve fine-grained segmentation maps in complex environments, which prohibits their implementation to real-world applications.To handle this problem. I plan to explore some promising algorithms for MSS. First, I will focus on improving skip connections of UNet by proposing a convolutional Transformer with cross-level interaction. Second, I will aim to break the shackle of model performance caused by the small number of annotations, through a shared-private architecture with adversarial multi-task learning to use as much additional data as possible.The potential impact of this project mainly includes two aspects: First, the methods studied in this project will improve the model performance of MSS in complex scenes and have strong potential to broaden the applicability of multi-modal/unlabeled/multi-task data. Second, this project will develop a generalized and instructive structure for MSS thanks to the shared-private mechanism and adversarial learning. It could be used for multi-tasks with heterogeneous inputs, with only a few modifications.Aims and ObjectivesThis project aims to address two key challenges to improve the performance and usability of MSS in complex surgical environments.Challenge 1: How to extract high-quality features and fuse them effectively?The latest MSS methods based on UNet fail to explore sufficient information from full scales due to the following two aspects:a) Not all connection pathways are effective due to the issue of semantic gaps in different layers. Those redundant and irrelevant connections increase the training difficulty of the network, even some can undermine the performance.b) The optimal combination of skip contributions is varied among different datasets, which depends on the scales and appearance of segmentation objects.To address the above problems, I consider replacing vanilla skip-connection pathways with Transformers to capture non-local features and perform effectively cross-level feature fusion.Challenge 2: How to utilize multi-modal data, unlabeled data, or even data from other tasks to improve the model performance?The scarcity of carefully-labelled datasets becomes an unavoidable limitation in DL-based MSS as both data and annotations are expensive to acquire. Previous methods pay less attention to utilizing different types of external data. Therefore, I consider using adversarial multi-task learning to build a uniform architecture for additional data, regardless of its type. This project will propose a novel cross-level convolutional Transformer for MSS to improve the skip-connection process of UNet.This project will propose a novel shared-private network with multiple encoders and decoders for MSS to utilize adversarial multi-task learning to handle additional data or tasks.As surgical image data are generally characterized by multiple modalities/scales and complex scenes, the research based on it could shed light on complex MSS.

该项目计划研究用于医学语义分割（MSS）的跨级卷积 Transformer 和对抗性多任务学习。 MSS 的目标是使用所表示内容的相应类别来标记图像的每个像素，以提供分割图。尽管基于深度学习的方法已经在 MSS 中实现了最先进的性能，但它们仍然难以在复杂环境中实现细粒度的分割图，这阻碍了它们在实际应用中的实现。我计划探索一些有前景的 MSS 算法。首先，我将通过提出具有跨级交互的卷积 Transformer 来重点改进 UNet 的跳跃连接。其次，我的目标是打破注释数量少造成的模型性能束缚，通过具有对抗性多任务学习的共享私有架构来使用尽可能多的额外数据。这个项目的潜在影响主要包括两个首先，本项目研究的方法将提高MSS在复杂场景下的模型性能，并且在拓宽多模态/无标签/多任务数据的适用性方面具有强大的潜力。其次，由于共享-私有机制和对抗性学习，该项目将为 MSS 开发一个通用且具有指导意义的结构。只需进行少量修改，即可用于具有异构输入的多任务。目的和目标该项目旨在解决两个关键挑战，以提高 MSS 在复杂手术环境中的性能和可用性。挑战 1：如何提取高质量的基于UNet的最新MSS方法由于以下两个方面而未能从全尺度探索足够的信息：a）由于不同层之间的语义间隙问题，并非所有连接路径都是有效的。这些冗余和不相关的连接增加了网络的训练难度，甚至有些会降低性能。b）不同数据集之间跳跃贡献的最佳组合是不同的，这取决于分割对象的尺度和外观。为了解决上述问题，我考虑用 Transformers 替代普通的跳连接路径，以捕获非局部特征并进行有效的跨级特征融合。挑战 2：如何利用多模态数据、无标签数据甚至其他任务的数据来改进模型性能？仔细标记的数据集的稀缺成为基于 DL 的 MSS 中不可避免的限制，因为获取数据和注释都很昂贵。以前的方法较少关注利用不同类型的外部数据。因此，我考虑使用对抗性多任务学习来为附加数据构建统一的架构，无论其类型如何。该项目将为 MSS 提出一种新型的跨级卷积 Transformer，以改进 UNet 的跳跃连接过程。该项目将为 MSS 提出一种具有多个编码器和解码器的新型共享私有网络，以利用对抗性多任务学习来处理额外的任务由于手术图像数据通常具有多模态/尺度和复杂场景的特点，因此基于其的研究可以为复杂的MSS提供线索。