CAREER: Rethinking Spiking Neural Networks from a Dynamical System Perspective

职业：从动态系统的角度重新思考尖峰神经网络

• 批准号: 2337646
• 负责人: Abhronil Sengupta
• 金额: $ 50万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-01-01 至 2028-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2337646&HistoricalAwards=false
• 关键词: CAREER; Rethinking; Spiking; Neural; Networks

Neuromorphic computing algorithms are emerging to be a disruptive paradigm driving machine learning research. Despite the significant energy savings enabled by such brain-inspired systems due to event-driven network operation, neuromorphic spiking neural networks (SNNs) remain largely limited to static vision tasks and convolutional architectures. Hence, there is an unmet need to revisit scalable SNN training algorithms from the ground-up by forging stronger correlations with bio-plausibility to leverage the enormous potential of time-based information processing and local learning capability of SNNs for sequential tasks. The project approaches spiking architectures as event-driven dynamical systems, wherein learning occurs through the convergence towards equilibrium states. The idea that neurons collectively adjust themselves to configurations (according to the sensory input being fed into a neural network system) such that they can better predict the input data has been a popular hypothesis. The collective neuron states can be interpreted as explanations of the input data. This compelling central idea provides motivation for this research and education program by pursuing two recently emerging methodologies for training neural architectures viz - Equilibrium Propagation (EP) and Implicit Differentiation on Equilibrium (IDE) that bear strong synergies with each other. The research has far-reaching impacts on Artificial Intelligence (AI) and the semiconductor industry, and on society at large, where disruptive computing paradigms like neuromorphic computing, emerging device technologies and cross-layer optimizations can potentially achieve significant improvements in data-intensive machine learning workloads in contrast to state-of-the-art approaches. The project will consider an integrated research, education and outreach plan that considers interdisciplinary curriculum development, graduate and undergraduate research mentoring, K-12 involvement, online educational module development and enhancing minority research participation to train the next generation of researchers and engineers jointly in the fields of Machine Learning and Nanoelectronics.The presented end-to-end research agenda has the potential of enabling a quantum leap in the efficiency of AI platforms by pursuing a multi-disciplinary perspective -- combining insights from machine learning and dynamical systems to hardware. The project spans complementary and inter-twined explorations across the following thrust areas: (1) Enabling local learning in SNNs for complex tasks by integrating EP with modern Hopfield networks to implement attention mechanisms, (2) Using IDE for developing a scalable and computationally efficient training method for Spiking Language Models, (3) Cross-layer software-hardware-application optimizations for efficient implementation of the algorithmic innovations on neuromorphic platforms for large-scale sequential learning tasks. The cross-layer nature of the project ranging from machine learning, dynamical system modelling, cutting edge AI applications and hardware design will serve as an ideal platform to pursue an interdisciplinary workforce development program. If successful, the research has the potential of developing scalable, robust, power and energy efficient neuromorphic computing paradigms that are applicable to a broad range of sequential processing tasks - a significant shift from the huge computational requirements of conventional deep learning solutions like Large Language Models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

神经形态计算算法已成为一种破坏性范式驱动机学习研究。尽管由于事件驱动的网络操作，这种受到大脑启发的系统启用了大量节能，但神经形态尖峰神经网络（SNN）仍然在很大程度上仅限于静态视觉任务和卷积架构。因此，通过与生物学知识性更强的相关性来利用基于时间的信息处理的巨大潜力和SNN的局部学习能力来实现SNNS的巨大潜力，从而从基础上重新审视可伸缩的SNN训练算法。该项目将尖峰体系结构作为事件驱动的动力学系统，其中学习是通过趋于平衡状态的融合进行的。神经元共同将自己调整为配置（根据被馈入神经网络系统的感觉输入）的想法，以便它们可以更好地预测输入数据是一个流行的假设。集体神经元状态可以解释为输入数据的解释。这一引人注目的中心思想通过追求两种训练神经体系结构的新兴方法来为这项研究和教育计划提供动力，即平衡传播（EP）以及对平衡（IDE）的隐性差异（IDE），这些方法具有强大的协同作用。这项研究对人工智能（AI）和半导体行业产生了深远的影响，以及整个社会的影响，在该社会中，诸如神经形态计算，新兴设备技术和跨层次优化的破坏性计算范式可能会在数据中有可能取得重大改进的机器学习工作。 The project will consider an integrated research, education and outreach plan that considers interdisciplinary curriculum development, graduate and undergraduate research mentoring, K-12 involvement, online educational module development and enhancing minority research participation to train the next generation of researchers and engineers jointly in the fields of Machine Learning and Nanoelectronics.The presented end-to-end research agenda has the potential of enabling a quantum leap in the efficiency of通过追求多学科的观点，AI平台结合了从机器学习和动态系统到硬件的见解。该项目跨越以下推力领域跨越互补和交叉的探索：（1）通过将EP与现代Hopfield网络集成以实施注意力机制，使SNN中的本地学习能够实现复杂的任务，（2）使用IDE来开发可扩展性和计算有效的培训，以实现跨越语言模型（3）跨层软件 - （3）跨层软件的实现，（3）用于大规模顺序学习任务的神经形态平台。该项目的跨层性质包括机器学习，动态系统建模，尖端AI应用程序和硬件设计，将成为追求跨学科劳动力发展计划的理想平台。如果成功的话，这项研究的潜力是开发适用于广泛的顺序处理任务的可扩展，稳健，能力和能源的神经形态计算范式 - 从大型深度语言模型（例如大型语言模型的巨大计算要求）的重大转变。该奖项（例如NSF的法定任务）反映了NSF的法定范围，反映了通过评估构成的构成群体的范围，并构成了构成群体的支持。

项目成果

SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation

SpikingBERT：使用隐式微分提炼 BERT 来训练尖峰语言模型

• DOI: 10.1609/aaai.v38i10.28975
• 发表时间: 2024
• 期刊: Proceedings of the AAAI Conference on Artificial Intelligence
• 作者: Bal, Malyaban;Sengupta, Abhronil
• 通讯作者: Sengupta, Abhronil