Cerebellum-inspired parallel deep learning

受小脑启发的并行深度学习

• 批准号: EP/X029336/1
• 负责人: Rui Ponte Costa
• 金额: $ 52.6万
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2024
• 资助国家: 英国
• 起止时间: 2024 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FX029336%2F1
• 关键词: Cerebellum; inspired; parallel; deep; learning

Deep learning, a type of machine learning, has recently undergone dramatic developments. It is already having an impact across society, from scientific discovery to climate change prediction. However, current deep learning models require billions of parameters which can take several weeks to train, costing millions of pounds with large carbon footprints. It is therefore becoming increasingly important to develop efficient training methods for deep neural networks.One of the key bottlenecks that underlie the inefficiency of deep neural networks is the need to perform a large number of computational steps sequentially. Here, inspired by recent findings on how biological neural networks learn, we propose to develop an efficient training algorithm for deep neural networks. We have recently proposed that a specialised brain region, the cerebellum, enables parallel learning in the brain. The cerebellum is defined by two features that are well placed to facilitate parallel learning: sparsity and modularity. First, the cerebellum contains highly sparse connectivity with only four input connections per neuron, which should result in faster learning. Second, the cerebellum is a highly modular system, which is well placed to enable parallel learning. Inspired by these cerebellar features we will develop a sparse-modular system for training deep learning networks capable of efficient parallel training. In collaboration with industry, the benefits of this approach will be demonstrated using a new type of parallel processor designed to accelerate machine learning. Overall, our work will lead to a novel approach to parallel deep learning, leading to a substantial reduction in training times and costs.

深度学习是一种机器学习，最近经历了戏剧性的发展。从科学发现到气候变化预测，它已经在整个社会上产生了影响。但是，当前的深度学习模型需要数十亿个参数，这可能需要数周的训练，耗资数百万英镑的碳足迹。因此，为深神经网络开发有效的训练方法变得越来越重要。基于深神经网络效率低下的关键瓶颈之一是需要顺序执行大量计算步骤。在这里，受到有关生物神经网络如何学习的最新发现的启发，我们建议为深神经网络开发有效的培训算法。我们最近提出，专门的大脑区域小脑可以使大脑平行学习。小脑的定义是两个特征，这些特征很好地促进并行学习：稀疏性和模块化。首先，小脑包含高度稀疏的连接性，每个神经元只有四个输入连接，这应该导致学习速度更快。其次，小脑是一个高度模块化的系统，可以很好地实现并行学习。受这些小脑功能的启发，我们将开发一个稀疏模块化系统，用于训练能够有效平行训练的深度学习网络。通过与行业合作，将使用旨在加速机器学习的新型并行处理器来证明这种方法的好处。总体而言，我们的工作将导致一种新颖的方法来平行深度学习，从而大大减少培训时间和成本。