NeTS: CSR: Medium: Collaborative Research: Enabling Flexible and High Performance Big Data Analytics Over Geo-Distributed Clouds

NeTS：CSR：中：协作研究：通过地理分布式云实现灵活且高性能的大数据分析

• 批准号: 1563095
• 负责人: Mosharaf Chowdhury
• 金额: $ 40万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-06-01 至 2020-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1563095&HistoricalAwards=false
• 关键词: NeTS; CSR; Medium; Collaborative; Research

Large organizations and small enterprises alike leverage datacenters across the globe to offer Internet services to their users. These sites routinely gather data pertaining to end user activities to provide better services, and they collect server monitoring logs and performance counters to ensure uninterrupted service. Although fast, efficient, and cost-effective analyses of these large datasets can significantly improve users' quality of experience and enable novel applications, the wide area network (WAN) that connects the datacenters poses a considerable challenge: because WAN bandwidth is limited and expensive, and WAN latency is high and variable, both the performance and timeliness of analytics are affected by the WAN.This project aims to build a new WAN-aware big data stack customized for flexible geo-distributed data analytics. The project will not impose any constraints on the set of queries that can be issued, and it will support a variety of performance objectives including obtaining timely responses, minimizing batch completion times, or using minimal bandwidth. To account for unpredictable and fine-timescale changes to WAN conditions and to enable coordination among the actions taken by different layers of the analytics stack, this project will enable holistic, cross-layer visibility and optimizations. It will incorporate awareness of the geo-distributed setting in the stack's upper layers (e.g., query optimization) and of application-level objectives in the lower layers (e.g., networking). This will result in a radical re-factoring of the API and interfaces between query optimization, query execution, resource negotiation, wide-area storage, and network routing/scheduling.Software artifacts from this project will be incorporated into existing open source big data stacks, making the research outcomes broadly available for public reuse. The experimental harnesses will be made available to ensure repeatability and to foster follow up research. The research outcomes will guide industry evolution as the industry slowly shifts from single-datacenter to geo-distributed settings. The project has a substantial educational component involving the introduction of new courses on big data systems at both graduate and undergraduate levels that will involve hands-on exercises with state-of-the-art big data software, and it will reach out to high-school students, women, and underrepresented minorities through big data boot camps.

大型组织和小型企业都利用全球的数据中心为其用户提供互联网服务。这些站点通常会收集与最终用户活动有关的数据，以提供更好的服务，并收集服务器监视日志和性能计数器，以确保不间断的服务。尽管对这些大型数据集的快速，高效和具有成本效益的分析可以显着提高用户的体验质量并启用新颖的应用程序，但连接数据中心的广域网络（WAN）构成了一个巨大的挑战：因为WAN带宽有限且昂贵，WAN潜伏期较高且可变，分析的性能和及时性都受WAN的影响。该项目旨在构建一个新的WAN-Aware Aware大数据堆栈，以定制用于灵活的地理分布数据分析。该项目将不会对可以发出的一系列查询施加任何限制，并且将支持各种绩效目标，包括获得及时的响应，最小化批次完成时间或使用最小的带宽。为了解释WAN条件的不可预测和微观的变化，并在不同层的分析堆栈采取的行动之间进行协调，该项目将使整体，跨层的可见性和优化。它将在堆栈的上层（例如查询优化）和下层中的应用级目标（例如网络）中纳入对地理分布的设置的认识。这将导致对查询优化，查询执行，资源谈判，广阔区域存储和网络路由/调度之间的API和接口的根本性重新分配。该项目的Software工件将被纳入现有的开源大数据堆栈中，使研究成果广泛可供公开重复使用。将提供实验线束以确保重复性并促进后续研究。研究成果将指导行业的发展，因为该行业从单人群慢慢转变为地理分布的设置。该项目具有实质性的教育部分，涉及在研究生和本科级别上引入有关大数据系统的新课程，这将涉及使用最先进的大数据软件的动手练习，并且它将接触到高级 - 学校的学生，女性和人数不足的少数民族通过大数据新兵训练营。