Collaborative Research: PPoSS: Planning: Cross-layer Coordination and Optimization for Scalable and Sparse Tensor Networks (CROSS)

合作研究：PPoSS：规划：可扩展和稀疏张量网络的跨层协调和优化（CROSS）

• 批准号: 2217086
• 负责人: Dong Li
• 金额: $ 6.25万
• 依托单位: University of California - Merced
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2024-02-29
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2217086&HistoricalAwards=false
• 关键词: Collaborative; Research; PPoSS; Planning; Cross

High-dimensional data computation or analytics are gaining importance in many domains, such as quantum chemistry/physics, quantum circuit simulation, brain processing, social networks, healthcare and machine/deep learning, to name a few. Tensors, a representation of high-dimensional data, are playing an increasingly critical role, and so are tensor methods. Tensor decompositions or factorizations of low-dimensional data (three to five dimensions) have been extensively studied over the past years from a high-performance computing and also compiler and computer architecture angles for their computational core operations, while tensor networks targeting very high-dimensional data (over ten dimensions) and extracting physically meaningful latent variables are underdeveloped because of their complicated mathematical nature, extremely high computational complexity, and more domain-dependent challenges. The project’s novelties are manifold: 1) memory heterogeneity-aware representations with algorithm and system optimizations, which could be adopted to solve other problems such as irregular applications and sparse numerical methods; 2) hardware-software co-design of specialized, sparse-tensor network-accelerator architectures, that are among the first hardware implementations of sparse-tensor networks. The project’s impacts are 1) advancing state-of-the-art tensor decomposition studies to model true higher-order and sparse data; 2) triggering a closer long-term collaboration ranging from academia to research labs to industry by studying solicitous applications; 3) bringing appropriate educational opportunities.This project proposes Cross-layer cooRdination and Optimization for Scalable and Sparse-Tensor Networks (CROSS) for heterogeneous systems that are equipped with various types of accelerators, such as GPUs, TPUs and FPGAs, as well as heterogeneous memories with dynamic and non-volatile random-access memories (DRAM+NVRAM). This research aims to study the sparsity in widely used tensor networks by introducing constraints, regularization, dictionaries, and/or domain knowledge for better data compression, faster computation, lower memory usage and better interpretability. Besides the sparsity challenges, sparse-tensor networks also suffer from the curse of dimensionality, aggravated data randomness and irregular program and memory access behaviors. This planning project conducts preliminary research that aims to address these challenges from four perspectives: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contraction (SpTC) algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse-tensor networks. The optimized sparse tensor networks will encompass efforts from high-performance computing, algorithms, compilers, computer architecture and performance modeling and will be tested under multiple application scenarios.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

高维数据计算或分析在许多领域中变得重要，例如量子化学/物理，量子电路模拟，大脑处理，社交网络，医疗保健和机器/深度学习，仅举几例。张量是高维数据的表示，起着越来越关键的作用，张量方法也是如此。在过去的几年中，从高性能计算，编译器和计算机架构角度来计算核心核心操作，在过去几年中，张量分解或因素化（三到五个维度）已被广泛研究，同时张张量的网络针对了非常高的量化（超过十分尺寸），因为它们具有较高的物理含量的量化，因为它们具有较高的量化量，因为它们的数量较高，因为它们具有较高的量化量，因为它们的数量较高，因为它们的数量较高，因为它们的数量较高，因为它们的量化量很高，因为它们的数量较高的量计算复杂性和更依赖域的挑战。该项目的新颖性是多种多样的：1）具有算法和系统优化的内存异质性表述，可以采用其他问题，例如不规则应用和稀疏数值方法； 2）硬件 - 软件软件共同设计的专用，稀疏网络加速器架构，这是稀疏张量网络的第一个硬件实现之一。该项目的影响是1）进步最新的张量分解研究，以模拟真正的高阶和稀疏数据； 2）通过研究坚固的应用来触发从学术界到研究实验室再到行业的更紧密的长期合作； 3）带来适当的教育机会。该项目提案跨层协调和优化，可扩展和稀疏量张量网络（交叉），用于异质系统，这些系统与各种类型的加速器，例如GPU，TPU和FPGA等各种类型的加速器，以及动态和非挥发性的随机记忆，以及+挥发性随机的记忆（这项研究旨在通过引入约束，调节，字典和/或域知识来研究广泛使用的张量网络中的稀疏性，以获得更好的数据压缩，更快的计算，较低的内存使用和更好的可解释性。除了稀疏挑战外，稀疏张量网络还遭受了维度，汇总数据随机性以及不规则程序和内存访问行为的诅咒。 This planning project conducts preliminary research that aims to address these challenges from four perspectives: (1) memory heterogeneity-aware representations and data (re-)arrangement, (2) balanced sparse tensor contract (SpTC) algorithms with smart page arrangement, (3) memoization and intelligent allocation to reduce computational cost, and (4) specialized accelerator architectures for sparse-tensor networks.优化的稀疏张量网络将涵盖高性能计算，算法，编译器，计算机架构和性能建模的工作，并将在多个应用程序场景下进行测试。该奖项反映了NSF的法定任务，并通过使用该基金会的智力功能和广泛的影响来评估NSF的法定任务。

项目成果

Merchandiser: Data Placement on Heterogeneous Memory for Task-Parallel HPC Applications with Load-Balance Awareness

• DOI: 10.1145/3572848.3577497
• 发表时间: 2023-02
• 期刊: Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming
• 作者: Zhen Xie;Jie Liu-;Jiajia Li;Dong Li