ASCENT: Collaborative Research: Scaling Distributed AI Systems based on Universal Optical I/O

ASCENT：协作研究：基于通用光学 I/O 扩展分布式人工智能系统

• 批准号: 2023468
• 负责人: Manya Ghobadi
• 金额: $ 32.5万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-15 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2023468&HistoricalAwards=false
• 关键词: ASCENT; Collaborative; Research; Scaling; Distributed

Our society is rapidly becoming reliant on neural networks based artificial intelligence computation. New algorithms are invented daily, increasing the memory and computational requirements for both inference and training. This explosive growth has created an enormous demand for distributed machine learning (ML) training and inference. Estimates by OpenAI illustrate the steady growth of computational requirements of 100x every two years since 2012, which is a 50x faster than the rate of computation improvements enabled previously through Moore’s Law of semiconductor industry that we have enjoyed in the last half-century. This new computation demand has been partly met by rapid development of hardware accelerators and software stacks to support these specialized computations. Hardware accelerators have provided a significant amount of speed-up but today’s training tasks can still take days and even weeks. The reason for this: as the number of workers (e.g. compute nodes) increases, the computation time per worker decreases, but the communication requirements between the nodes increase, creating a bottleneck in the interconnect between the compute nodes. Future distributed ML systems will require 1-2 orders of magnitude higher interconnect bandwidth per node, creating a pressing need for entirely new ways to build interconnects for distributed ML systems. This proposal aims to create a new paradigm for scaling distributed ML computation, by developing a scalable interconnect solution based on advancing the integrated electronics and photonics technology that enables direct node-to-node optical fiber connectivity. The proposed cross-stack collaborative multi-disciplinary work will enable the education and training of a unique crop of engineers and scientists that cross the boundaries of machine learning, networking, and electronic-photonic systems and devices, which are in severe demand. The principal investigators have an established track record of direct engagement with high-school students providing summer internships at Berkeley Wireless Research Center and MIT’s Women’s Technology Program, as well as exemplary undergraduate research activities at Boston University. The educational and outreach activities the PIs have put in place will ensure early exposure and continued training of new generation of leaders in this field, from K-12, through undergraduate and graduate studies, and continuing workforce education, with special focus on underrepresented students.The interconnect has emerged as the key bottleneck in enabling the full potential of distributed ML. Future ML workloads are likely to require tens of Tbps of bandwidth per device. Ubiquitous deployment of logically-connected, physically distributed computation across shelf, rack and row scale can only be enabled by a new universal I/O that enables socket to socket communication at the energy, latency and bandwidth density of in-package interconnects. No such technology currently exists. Silicon-photonics based optical I/O has the potential to address this critical challenge, but fundamental advances–from chip manufacturing to routing algorithms–are still needed to ensure the scalability of these interconnect systems. To enable high-bandwidth density and energy-efficiency, dense wavelength division multiplexing must be used. High-efficiency ring resonator-based modulators and comb laser sources are needed to enable Tbps rates over each fiber connection and socket bandwidth scaling from 10s to 100s of Tbps. New link architectures like the proposed laser-forwarded coherent link are needed to enable high-efficiency external centralized comb laser sources with modest (sub-mW) power per wavelength per fiber port. The proposed work will also develop new scheduling algorithms, network architectures, and workload parallelism strategy to leverage the bandwidth density and low-latency of the universal optical I/O, to map large AI workloads with massive datasets to a scalable distributed compute system.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们的社会正在迅速与基于神经网络的人工智能计算相关。每天都会发明新算法，从而增加了推理和培训的记忆和计算要求。这种爆炸性的增长创造了对分布式机器学习（ML）培训和推理的增强需求。 Openai的估计说明，自2012年以来，计算需求的稳定增长每两年每两年100倍，这比以前通过摩尔的半导体行业定律所享有的计算率提高了50倍，而我们在上半个世纪中享有的。通过快速开发硬件加速器和软件堆栈以支持这些专业计算，这一新计算需求得到了部分满足。硬件加速器提供了大量加速，但是当今的培训任务仍然可能需要几天甚至数周。原因是：随着工人数量（例如计算节点）的增加，每个工人的计算时间减少，但是节点之间的通信要求增加，从而在计算节点之间的互连中产生了瓶颈。未来的分布式ML系统将需要每个节点的互连带宽较高1-2个数量级，从而迫切需要全新的方法来构建分布式ML系统的互连。该建议旨在通过基于推进集成电子和光子技术技术来开发可扩展的互连解决方案来创建一个新的范式来扩展分布式ML计算，该解决方案可以启用直接节点 - 节点光纤连接性。拟议的跨堆栈协作多学科工作将使跨越机器学习，网络，电子 - 光电系统和设备的界限的独特工程师和科学家的教育和培训，这些界限迫在眉睫。首席调查人员拥有与高中生直接互动的既定记录，该学生在伯克利无线研究中心和麻省理工学院的女性技术计划以及波士顿大学的典范本质研究活动中提供暑期实习。 PIS进行的教育和外展活动将确保早期接触并继续培训该领域的新一代领导者，从K-12，通过本科和研究生学习以及继续进行劳动力教育，并特别关注代表性不足的学生。互连已成为启用分布式ML的全部潜力的关键瓶颈。未来的ML工作负载可能需要每台设备几十TBP的带宽。在逻辑连接，物理分布的跨架子，机架和行尺度上的无处不在的部署只能通过新的通用I/O来启用，该通用I/O可以使套接字在包装中互连的能量，延迟和带宽密度以插座通信。目前没有这样的技术。基于硅 - 光子学的光学I/O具有应对这一关键挑战的潜力，但是从芯片制造到路由算法的基本进展仍然需要确保这些互连系统的可扩展性。为了实现高带宽密度和能效，必须使用致密波长的多路复用。基于高效率的环谐振器调节器和梳子激光源可以使每个光纤连接的TBPS速率和插座带宽从10s到100s的TBPS缩放。需要新的链接体系结构，例如所提出的激光反向连贯链接，以使每个波长每个波长每个波长具有适中的（子-MW）功率的高效外部集中式梳子激光源。拟议的工作还将开发新的时间表算法，网络架构和工作量并行策略，以利用通用光学I/O的带宽密度和低延迟的带宽密度和低延迟，以将大量的AI工作负载绘制为大型AI工作负载，以大规模的数据集为可伸缩的分布式系统，这些奖项通过评估NSF的合法任务和支持的良好的支持。 标准。

项目成果

Emerging Optical Interconnects for AI Systems

适用于人工智能系统的新兴光互连

• DOI: 10.1364/ofc.2022.th1g.1
• 发表时间: 2022
• 期刊: OFC
• 作者: Ghobadi, Manya

TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

• 发表时间: 2022-02
• 作者: Weiyang Wang;Moein Khazraee;Zhizhen Zhong;M. Ghobadi;Zhihao Jia;Dheevatsa Mudigere;Ying Zhang

Demonstration of WDM-Enabled Ultralow-Energy Photonic Edge Computing

支持 WDM 的超低能量光子边缘计算演示

• DOI: 10.1364/ofc.2022.th3a.3
• 发表时间: 2022
• 期刊: OFC
• 作者: Sludds, Alexander;Hamerly, Ryan;Bandyopadhyay, Saumil;Zhong, Zhizhen;Chen, Zaijun;Bernstein, Liane;Ghobadi, Manya;Englund, Dirk
• 通讯作者: Englund, Dirk

SiP-ML: high-bandwidth optical network interconnects for machine learning training

• DOI: 10.1145/3452296.3472900
• 发表时间: 2021-08
• 期刊: Proceedings of the 2021 ACM SIGCOMM 2021 Conference
• 作者: Mehrdad Khani Shirkoohi;M. Ghobadi;M. Alizadeh;Ziyi Zhu;M. Glick;K. Bergman;A. Vahdat;Benjamin Klenk-Ben