CL-LSG: Continual Learning via Learnable Sparse Growth

Continual learning (CL) has been developed to learn new tasks sequentially and perform knowledge transfer from the old tasks to the new ones without forgetting, which is well known as catastrophic forgetting . While recent structure-based learning methods show the capability of alleviating the forgetting problem, these methods require a complex learning process to gradually grow-and-prune of a full-size network for each task, which is inefficient. To address this problem and enable efficient network expansion for new tasks, to the best of our knowledge, we are the first to develop a learnable sparse growth (LSG) method, which explicitly optimizes the model growth to only select important and necessary channels for growing. Building on the LSG, we then propose CL-LSG , a novel end-to-end CL framework to grow the model for each new task dynamically and sparsely. Different from all previous structure-based CL methods that start from and then prune (i.e., two-step) a full-size network, our framework starts from a compact seed network with a much smaller size and grows to the necessary model size (i.e., one-step) for each task, which eliminates the need of additional pruning in previous structure-based growing methods.

持续学习（CL）旨在按顺序学习新任务，并在不遗忘的情况下将知识从旧任务转移到新任务，这也就是众所周知的灾难性遗忘问题。虽然近期基于结构的学习方法显示出缓解遗忘问题的能力，但这些方法需要一个复杂的学习过程，针对每个任务对一个完整规模的网络逐步进行增长和修剪，效率低下。为了解决这个问题并实现针对新任务的高效网络扩展，据我们所知，我们率先开发了一种可学习的稀疏增长（LSG）方法，该方法明确地优化模型增长，仅选择重要且必要的通道进行增长。在此基础上，我们进而提出了CL - LSG，这是一种新颖的端到端持续学习框架，能够动态且稀疏地为每个新任务增长模型。与之前所有基于结构的持续学习方法不同，那些方法从一个完整规模的网络开始然后进行修剪（即两步法），我们的框架从一个尺寸小得多的紧凑种子网络开始，并针对每个任务增长到必要的模型尺寸（即一步法），这就消除了之前基于结构的增长方法中额外修剪的需要。