CDS&E: Efficient and Robust Recurrent Neural Networks

CDS

• 批准号: 1821144
• 负责人: Qiang Ye
• 金额: $ 20万
• 依托单位: University of Kentucky Research Foundation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1821144&HistoricalAwards=false
• 关键词: CDS& Efficient; Robust; Recurrent; Neural

Deep neural networks have emerged over the last decade as one of the most powerful machine learning methods. Recurrent neural networks (RNNs) are special neural networks that are designed to efficiently model sequential data such as speech and text data by exploiting temporal connections within a sequence and handling varying sequence lengths in a dataset. While RNN and its variants have found success in many real-world applications, there are various issues that make them difficult to use in practice. This project will systematically address some of these difficulties and develop an efficient and robust RNN. Computer codes derived in this project will be made freely available. The research results will have applications in a variety of areas involving sequential data learning, including computer vision, speech recognition, natural language processing, financial data analysis, and bioinformatics.As in other neural networks, training of RNNs typically involves some variants of gradient descent optimization, which is prone to so-called vanishing or exploding gradient problems. Regularization of RNNs, which refers to techniques used to prevent the model from overfitting the raining data and hence poor generalization to new data, is also challenging. The current preferred RNN architectures such as the Long-Short-Term-Memory networks have highly complex structures with numerous additional interacting elements that are not easy to understand. This project develops an RNN that extends a recent orthogonal/unitary RNNs to more effectively model long and short term dependency of sequential data. Through an indirect parametrization of recurrent matrix, dropout regularization techniques will be developed. The network developed in this project will retain the simplicity and efficiency of basic RNNs but enhance some key capabilities for robust applications. In particular, the project will include a study of applications of RNNs to some bioinformatics problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络在过去十年中已成为最强大的机器学习方法之一。循环神经网络 (RNN) 是一种特殊的神经网络，旨在通过利用序列内的时间连接并处理数据集中不同的序列长度来有效地对序列数据（例如语音和文本数据）进行建模。虽然 RNN 及其变体在许多实际应用中取得了成功，但存在各种问题使其难以在实践中使用。该项目将系统地解决其中一些困难，并开发一个高效且强大的 RNN。该项目中衍生的计算机代码将免费提供。研究成果将在涉及顺序数据学习的各种领域得到应用，包括计算机视觉、语音识别、自然语言处理、金融数据分析和生物信息学。与其他神经网络一样，RNN 的训练通常涉及梯度下降的一些变体优化，这很容易出现所谓的梯度消失或爆炸问题。 RNN 的正则化也具有挑战性，它是指用于防止模型过度拟合下雨数据并因此对新数据泛化不良的技术。当前首选的 RNN 架构（例如长短期记忆网络）具有高度复杂的结构，具有许多不易理解的额外交互元素。该项目开发了一种 RNN，它扩展了最近的正交/单一 RNN，以更有效地对序列数据的长期和短期依赖性进行建模。通过循环矩阵的间接参数化，将开发丢失正则化技术。该项目开发的网络将保留基本 RNN 的简单性和效率，但增强了健壮应用程序的一些关键功能。特别是，该项目将包括 RNN 在某些生物信息学问题上的应用研究。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

AUTM Flow: Atomic Unrestricted Time Machine for Monotonic Normalizing Flows

AUTM Flow：用于单调归一化流的原子无限制时间机

• 发表时间: 2022-01
• 期刊: Uncertainty in artificial intelligence
• 作者: Cai, D.;Ji, Y.;He, H.;Ye, Q.
• 通讯作者: Ye, Q.

Adaptive Weighted Discriminator for Training Generative Adversarial Networks

用于训练生成对抗网络的自适应加权判别器

• DOI: 10.1109/cvpr46437.2021.00475
• 发表时间: 2021-06
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Zadorozhnyy V;Cheng Q;Ye Q
• 通讯作者: Ye Q

Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory

用于短期记忆的特征值归一化循环神经网络

• DOI: 10.1609/aaai.v34i04.5831
• 发表时间: 2020-06
• 期刊: Proceedings of the AAAI Conference on Artificial Intelligence
• 作者: Helfrich, Kyle;Ye, Qiang
• 通讯作者: Ye, Qiang

Batch Normalization Preconditioning for Neural Network Training

神经网络训练的批量归一化预处理

• 发表时间: 2021-08-02
• 期刊: ArXiv
• 作者: Susanna Lange;Kyle E. Helfrich;Qiang Ye
• 通讯作者: Qiang Ye

Improving RNA secondary structure prediction via state inference with deep recurrent neural networks

通过深度循环神经网络的状态推断改进 RNA 二级结构预测

• DOI: 10.1515/cmb-2020-0002
• 发表时间: 2020-03
• 期刊: Computational and Mathematical Biophysics
• 作者: Willmott, Devin;Murrugarra, David;Ye, Qiang
• 通讯作者: Ye, Qiang