CIF: Small: Deep Stochasticity for Private Collaborative Deep Learning

CIF：小：私人协作深度学习的深度随机性

• 批准号: 2215088
• 负责人: Yongqiang Wang
• 金额: $ 35万
• 依托单位: Clemson University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2215088&HistoricalAwards=false
• 关键词: CIF; Small; Deep; Stochasticity; Private

By now, Deep Learning is achieving unprecedented performance levels in many applications ranging from computer vision to natural language processing to drug design. Training models usually require large volumes of training data, said data being collected from multiple individuals/organizations to ensure heterogeneity since homogeneous data may lead to over-fitting. Training data often contain sensitive information, e.g., healthcare records, browsing history, or financial transactions, thereby posing privacy threats for the individuals from whom the data were collected. Although multi-machine collaborative learning, such as decentralized learning and federated learning, allegedly solves privacy concerns by never letting the raw training data leave the participating machines, recent studies have revealed a completely different picture: Not only can features of the training data be inferred from shared gradient/model updates, but even the raw data can be reversely inferred from these shared gradients. Moreover, adding noise to shared gradients, a de facto standard for achieving differential privacy, becomes effective only when the noise is sufficiently large, possibly leading to a degradation of the training accuracy. This project, instead, seeks to enable privacy protection for participating machines through judicious randomization that exploits the structure of collaborative learning algorithms and leverages their natural resiliency to error. The project will enrich the current curriculum by providing new modules on privacy-preserving decentralized learning for both undergraduate and graduate classes. Broadening Participation in Computing will be addressed through outreach activities involving minority students via Clemson PEER (Programs for Educational Enrichment and Retention) and WISE (Women in Science and Engineering). The project explores several different approaches to judiciously embed stochasticity at the algorithmic level, so-called deep stochasticity, in order to enable privacy protection in the collaborative learning process. The proposed approach exploits the natural resiliency of deep learning algorithms to parameter errors/noises, and enables privacy without compromising accuracy or incurring heavy computation/communication overheads with the flexibility to accommodate additional mechanisms like cryptography. The techniques are applicable to both parameter-server-free decentralized learning and to parameter-server facilitated federated learning. The main research thrusts center on the design of collaborative learning algorithms that use stochastic quantization schemes for inter-machine communications and random learning stepsizes in building the iterates. Rigorous analysis frameworks will be developed to quantitatively evaluate the strength of the privacy protection being achieved, and the theoretical results will be systematically validated through experiments with robot networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

到目前为止，深度学习在从计算机视觉到自然语言处理再到药物设计的许多应用中都达到了前所未有的性能水平。训练模型通常需要大量的训练数据，这些数据是从多个个人/组织收集的，以确保异质性，因为同质数据可能会导致过度拟合。训练数据通常包含敏感信息，例如医疗记录、浏览历史或金融交易，从而对收集数据的个人构成隐私威胁。尽管多机器协作学习，例如去中心化学习和联邦学习，据称通过不让原始训练数据离开参与机器来解决隐私问题，但最近的研究揭示了完全不同的情况：不仅可以推断训练数据的特征来自共享梯度/模型更新，但即使是原始数据也可以从这些共享梯度反向推断。此外，向共享梯度添加噪声是实现差分隐私的事实上的标准，只有当噪声足够大时才有效，可能会导致训练精度下降。相反，该项目寻求通过明智的随机化来实现参与机器的隐私保护，该随机化利用协作学习算法的结构并利用其对错误的自然弹性。该项目将为本科生和研究生课程提供有关保护隐私的去中心化学习的新模块，从而丰富当前的课程。扩大对计算机的参与将通过克莱姆森 PEER（教育丰富和保留计划）和 WISE（科学与工程领域的女性）涉及少数族裔学生的外展活动来解决。该项目探索了几种不同的方法，在算法层面明智地嵌入随机性，即所谓的深度随机性，以便在协作学习过程中实现隐私保护。所提出的方法利用深度学习算法对参数错误/噪声的自然弹性，并在不影响准确性或招致大量计算/通信开销的情况下实现隐私，并且可以灵活地适应密码学等附加机制。这些技术既适用于无参数服务器的分散学习，也适用于参数服务器促进的联邦学习。主要研究重点是协作学习算法的设计，该算法使用随机量化方案进行机器间通信，并在构建迭代时使用随机学习步长。将开发严格的分析框架来定量评估所实现的隐私保护的强度，并通过机器人网络实验系统地验证理论结果。该奖项反映了 NSF 的法定使命，并通过使用基金会的评估认为值得支持。智力价值和更广泛的影响审查标准。

项目成果

Ensuring both Almost Sure Convergence and Differential Privacy in Nash Equilibrium Seeking on Directed Graphs

确保有向图纳什均衡搜索中的几乎确定收敛和差分隐私

• DOI: 10.1109/tac.2024.3366037
• 发表时间: 2024-01
• 期刊: IEEE Transactions on Automatic Control
• 影响因子: 6.8
• 作者: Wang, Yongqiang;Başar, Tamer
• 通讯作者: Başar, Tamer

Tailoring Gradient Methods for Differentially-Private Distributed Optimization

用于差分隐私分布式优化的定制梯度方法

• DOI: 10.1109/tac.2023.3272968
• 发表时间: 2023-01
• 期刊: IEEE Transactions on Automatic Control
• 影响因子: 6.8
• 作者: Wang, Yongqiang;Nedić, Angelia
• 通讯作者: Nedić, Angelia

Differentially-private Distributed Algorithms for Aggregative Games with Guaranteed Convergence

保证收敛的聚合博弈的差分私有分布式算法

• DOI: 10.1109/tac.2024.3351068
• 发表时间: 2024-01
• 期刊: IEEE Transactions on Automatic Control
• 影响因子: 6.8
• 作者: Wang, Yongqiang;Nedić, Angelia
• 通讯作者: Nedić, Angelia

Differentially-Private Distributed Optimization with Guaranteed Optimality

保证最优性的差分私有分布式优化

• DOI: 10.1109/cdc49753.2023.10383285
• 发表时间: 2023-12
• 期刊: IEEE
• 作者: Wang, Yongqiang;Nedić, Angelia
• 通讯作者: Nedić, Angelia

Quantization Avoids Saddle Points in Distributed Optimization

量化避免分布式优化中的鞍点

• 发表时间: 2024-03
• 期刊: Proceedings of the National Academy of Sciences of the United States of America
• 影响因子: 11.1
• 作者: Yanan Bo; Yongqiang Wang
• 通讯作者: Yongqiang Wang