ASCENT: Collaborative Research: Scaling Distributed AI Systems based on Universal Optical I/O

ASCENT：协作研究：基于通用光学 I/O 扩展分布式人工智能系统

• 批准号: 2023468
• 负责人: Manya Ghobadi
• 金额: $ 32.5万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-15 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2023468&HistoricalAwards=false
• 关键词: ASCENT; Collaborative; Research; Scaling; Distributed

Our society is rapidly becoming reliant on neural networks based artificial intelligence computation. New algorithms are invented daily, increasing the memory and computational requirements for both inference and training. This explosive growth has created an enormous demand for distributed machine learning (ML) training and inference. Estimates by OpenAI illustrate the steady growth of computational requirements of 100x every two years since 2012, which is a 50x faster than the rate of computation improvements enabled previously through Moore’s Law of semiconductor industry that we have enjoyed in the last half-century. This new computation demand has been partly met by rapid development of hardware accelerators and software stacks to support these specialized computations. Hardware accelerators have provided a significant amount of speed-up but today’s training tasks can still take days and even weeks. The reason for this: as the number of workers (e.g. compute nodes) increases, the computation time per worker decreases, but the communication requirements between the nodes increase, creating a bottleneck in the interconnect between the compute nodes. Future distributed ML systems will require 1-2 orders of magnitude higher interconnect bandwidth per node, creating a pressing need for entirely new ways to build interconnects for distributed ML systems. This proposal aims to create a new paradigm for scaling distributed ML computation, by developing a scalable interconnect solution based on advancing the integrated electronics and photonics technology that enables direct node-to-node optical fiber connectivity. The proposed cross-stack collaborative multi-disciplinary work will enable the education and training of a unique crop of engineers and scientists that cross the boundaries of machine learning, networking, and electronic-photonic systems and devices, which are in severe demand. The principal investigators have an established track record of direct engagement with high-school students providing summer internships at Berkeley Wireless Research Center and MIT’s Women’s Technology Program, as well as exemplary undergraduate research activities at Boston University. The educational and outreach activities the PIs have put in place will ensure early exposure and continued training of new generation of leaders in this field, from K-12, through undergraduate and graduate studies, and continuing workforce education, with special focus on underrepresented students.The interconnect has emerged as the key bottleneck in enabling the full potential of distributed ML. Future ML workloads are likely to require tens of Tbps of bandwidth per device. Ubiquitous deployment of logically-connected, physically distributed computation across shelf, rack and row scale can only be enabled by a new universal I/O that enables socket to socket communication at the energy, latency and bandwidth density of in-package interconnects. No such technology currently exists. Silicon-photonics based optical I/O has the potential to address this critical challenge, but fundamental advances–from chip manufacturing to routing algorithms–are still needed to ensure the scalability of these interconnect systems. To enable high-bandwidth density and energy-efficiency, dense wavelength division multiplexing must be used. High-efficiency ring resonator-based modulators and comb laser sources are needed to enable Tbps rates over each fiber connection and socket bandwidth scaling from 10s to 100s of Tbps. New link architectures like the proposed laser-forwarded coherent link are needed to enable high-efficiency external centralized comb laser sources with modest (sub-mW) power per wavelength per fiber port. The proposed work will also develop new scheduling algorithms, network architectures, and workload parallelism strategy to leverage the bandwidth density and low-latency of the universal optical I/O, to map large AI workloads with massive datasets to a scalable distributed compute system.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我的社会迅速依赖基于神经网络的人工智能计算。自2012年以来，每两年的要求提高了100倍，我们在过去半个世纪中享受的摩尔的半产物行业定律以前。提供了重要的，但今天的任务仍然可以拖延几周：随着工人的数量（例如计算节点）增加，计算时间降低，但节点S之间的共处要求增加，在互连中增加计算节点。每个节点较高的互连带宽，创造了用于分布式ML系统的全新方法，旨在创建一个新的范式节点到节点的光纤连接。与Hig H-School学生的直接互动记录在伯克利研究和MIT的女性技术计划的暑期实习期间从K-12到本科和研究生学习，以及互联的劳动力教育，特别关注代表性不足的学生。互连已成为主要的瓶颈，以使分布式ML的全部潜力可能需要数十个TBP。设备的架子，架子和行刻度只能通过能量的新通用I/O套接字来启用。关键挑战的地址，但基本进展 - 从芯片制造到路由算法 - 逐渐降低这些互连系统的ty，以使基于效率的调制器和梳子源是效率的高带宽和能量效率的调节器。每个纤维连接和套接字带宽将ROM缩放10s到100 s的TBP，例如支撑激光的相干链接，以使高效的外部集中式梳子激光源具有每个波长每个波长每个波长。

项目成果

Emerging Optical Interconnects for AI Systems

适用于人工智能系统的新兴光互连

• DOI: 10.1364/ofc.2022.th1g.1
• 发表时间: 2022
• 期刊: OFC
• 作者: Ghobadi, Manya

TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

• 发表时间: 2022-02
• 作者: Weiyang Wang;Moein Khazraee;Zhizhen Zhong;M. Ghobadi;Zhihao Jia;Dheevatsa Mudigere;Ying Zhang

Demonstration of WDM-Enabled Ultralow-Energy Photonic Edge Computing

支持 WDM 的超低能量光子边缘计算演示

• DOI: 10.1364/ofc.2022.th3a.3
• 发表时间: 2022
• 期刊: OFC
• 作者: Sludds, Alexander;Hamerly, Ryan;Bandyopadhyay, Saumil;Zhong, Zhizhen;Chen, Zaijun;Bernstein, Liane;Ghobadi, Manya;Englund, Dirk
• 通讯作者: Englund, Dirk

SiP-ML: high-bandwidth optical network interconnects for machine learning training

• DOI: 10.1145/3452296.3472900
• 发表时间: 2021-08
• 期刊: Proceedings of the 2021 ACM SIGCOMM 2021 Conference
• 作者: Mehrdad Khani Shirkoohi;M. Ghobadi;M. Alizadeh;Ziyi Zhu;M. Glick;K. Bergman;A. Vahdat;Benjamin Klenk-Ben