CSR: Medium: Collaborative Research: Programming parallel in-memory data-center applications with Piccolo

CSR：媒介：协作研究：使用 Piccolo 对并行内存数据中心应用程序进行编程

• 批准号: 1065169
• 负责人: Jinyang Li
• 金额: $ 52.33万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-07-01 至 2015-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1065169&HistoricalAwards=false
• 关键词: CSR; Medium; Collaborative; Research; Programming

There is a rising demand to scale application performance by distributingcomputation across many machines in a data-center. It is difficult to writeefficient and robust parallel programs in the data-center setting because programmers need to worry about reducing communication overhead while handling possible machine failures. This project investigates a new data-centric parallel programmingmodel, called Piccolo, that can simplify the construction of in-memorydata-center applications such as PageRank, neural network training etc. In-memory applications can hold all their intermediate states in the aggregatememory of many machines and benefit from sharing these intermediate statesbetween machines during computation. Traditionally, these applicationshave been built using low-level communication-centric primitives such as MPI,resulting in significant programming complexity. The recently popular MapReduce and Dryad also do not fit well with these applicationsbecause their data flow programming model lacks support for shared states.Unlike data flow models, Piccolo explicitly supports the sharing of mutable,distributed states via a key/value table interface. Piccolo makes sharingefficient by optimizing for locality of access to shared tables andautomatically resolving write-write conflicts using user-defined accumulationfunctions. As a result, Piccolo is easy to program for, enables applicationsthat do not fit into MapReduce, and achieves good scalable performance.

通过在数据中心的多台机器上分配计算来扩展应用程序性能的需求不断增长。在数据中心环境中编写高效且健壮的并行程序很困难，因为程序员需要担心在处理可能的机器故障时减少通信开销。 该项目研究了一种新的以数据为中心的并行编程模型，称为 Piccolo，它可以简化内存数据中心应用程序的构建，例如 PageRank、神经网络训练等。内存应用程序可以将其所有中间状态保存在许多内存的聚合内存中。机器并受益于在计算期间在机器之间共享这些中间状态。 传统上，这些应用程序是使用 MPI 等以低级通信为中心的原语构建的，导致编程复杂性很高。最近流行的 MapReduce 和 Dryad 也不太适合这些应用程序，因为它们的数据流编程模型缺乏对共享状态的支持。与数据流模型不同，Piccolo 明确支持通过键/值表接口共享可变的分布式状态。 Piccolo 通过优化共享表访问的局部性并使用用户定义的累积函数自动解决写入冲突，从而提高共享效率。 因此，Piccolo 易于编程，支持不适合 MapReduce 的应用程序，并实现良好的可扩展性能。