Intelligent Management of Big Data Storage

大数据存储智能管理

• 批准号: EP/L00738X/1
• 负责人: Peter Harrison
• 金额: $ 46.9万
• 依托单位: Imperial College London
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2014
• 资助国家: 英国
• 起止时间: 2014 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FL00738X%2F1
• 关键词: Intelligent; Management; Big; Data; Storage

The continuing revolutionary growth of data volumes and the increasing diversity of data-intensive applications demands an urgent investigation of effective means for efficient storage management. In the summer of 2012, the volume of data in the world was around 10 to the power of 21 bytes, about 1.1TB per internet user, and this volume continues to increase at about 50% Compound Annual Growth Rate. It has been said that "By 2013, storage systems will no longer be manually tunable for performance or manual data placement. Similar to virtual memory management, the storage array's algorithms will determine data placement (The Future of Storage Management, Gartner 2010). Meeting service-level objective/agreement (SLO/SLA) requirements for data-intensive applications is not straightforward and will become increasingly more challenging. In particular, there is an increasing need for intelligent mechanisms to manage the underlying architectures' infrastructure, taking into account the advent of new device technologies.To cope with this challenge, we propose a research program in the mainstream of EPSRC's theme "Towards an intelligent information infrastructure (TI3)", specifically with reference to the "deluge of data" and the exploration of "emerging technologies for low power, high speed, high density, low cost memory and storage solutions". Today, with the widespread distribution of storage, for example in cloud storage solutions, it is difficult for an infrastructure provider to decide where data resides, on what type of device, co-located with what other data owned by which other (maybe competing) user, and even in what country. The need to meet energy-consumption targets compounds this problem. These decisional problems motivate the present research proposal, which aims at developing new model-based techniques and algorithms to facilitate the effective administration of data-intensive applications and their underlying storage device infrastructure.We propose to develop techniques and tools for the quantitative analysis and optimisation of multi-tiered data storage systems. The primary objective is to develop novel modelling approaches to define and facilitate the most appropriate data placement and data migration strategies. These strategies share the common aim of placing data on the most effective target device in a tiered storage architecture. In the proposed research, the allocation algorithm will be able to decide the placement strategy and trigger data migrations to optimize an appropriate utility function. Our research will also take into account the likely quantitative impact of evolving storage and energy-efficiency technologies, by developing suitable models of these and integrating them into our tier-allocation methodologies. In essence, our models will be specialised for different storage and power technologies (e.g. fossil fuel, solar, wind). The models, optimisers and methodologies that we produce will be tested in pilot implementations on our in-house cloud (already purchased); on Amazon EC2 resources; and finally in an industrial, controlled production environment as part of our collaboration with NetApp. This will provide feedback to enable us to refine, enhance and extend our techniques, and hence to further improve the utility of the biggest of storage systems.

数据量的持续革命性增长和数据密集型应用程序日益多样化，迫切需要研究高效存储管理的有效手段。 2012年夏天，全球数据量约为10的21字节次方，每个互联网用户约为1.1TB，并且这一数据量继续以约50%的复合年增长率增长。有人说，“到 2013 年，存储系统将不再能够手动调节性能或手动数据放置。与虚拟内存管理类似，存储阵列的算法将决定数据放置（存储管理的未来，Gartner 2010）。会议数据密集型应用程序的服务级别目标/协议（SLO/SLA）要求并不简单，并且将变得越来越具有挑战性，特别是，考虑到管理底层架构基础设施的智能机制的需求不断增加。为了应对这一挑战，我们提出了EPSRC主题“迈向智能信息基础设施（TI3）”的主流研究计划，特别针对“数据洪流”和“数据洪流”的探索低功耗、高速、高密度、低成本内存和存储解决方案的新兴技术”。如今，随着存储的广泛分布，例如在云存储解决方案中，基础设施提供商很难决定数据驻留在何处什么类型的设备，与其他（可能是竞争的）用户拥有的其他数据位于同一位置，甚至位于哪个国家/地区。满足能源消耗目标的需要使这个问题变得更加复杂。这些决策问题激发了当前的研究提案，该提案旨在开发新的基于模型的技术和算法，以促进数据密集型应用程序及其底层存储设备基础设施的有效管理。我们建议开发用于定量分析和优化的技术和工具多层数据存储系统。主要目标是开发新颖的建模方法来定义和促进最合适的数据放置和数据迁移策略。这些策略的共同目标是将数据放置在分层存储架构中最有效的目标设备上。在所提出的研究中，分配算法将能够决定放置策略并触发数据迁移以优化适当的效用函数。我们的研究还将考虑不断发展的存储和能源效率技术可能产生的定量影响，通过开发合适的模型并将其集成到我们的分层分配方法中。本质上，我们的模型将专门针对不同的存储和电力技术（例如化石燃料、太阳能、风能）。我们生产的模型、优化器和方法将在我们的内部云（已购买）上进行试点实施测试； Amazon EC2 资源；最后，作为我们与 NetApp 合作的一部分，在工业受控生产环境中进行。这将提供反馈，使我们能够改进、增强和扩展我们的技术，从而进一步提高最大存储系统的实用性。

项目成果

QRF An Optimization-Based Framework for Evaluating Complex Stochastic Networks

QRF 用于评估复杂随机网络的基于优化的框架

• DOI: 10.1145/2724709
• 发表时间: 2016
• 期刊: ACM Transactions on Modeling and Computer Simulation
• 影响因子: 0.9
• 作者: Casale G

Performance-Energy Trade-offs in Smartphones

智能手机的性能与能耗权衡

• DOI: 10.1145/2988287.2989140
• 发表时间: 2016
• 作者: Chis T

Analyzing Replacement Policies in List-Based Caches with Non-Uniform Access Costs

• DOI: 10.1109/infocom.2018.8485894
• 发表时间: 2018-04
• 期刊: IEEE INFOCOM 2018 - IEEE Conference on Computer Communications
• 作者: G. Casale

Accelerating Performance Inference over Closed Systems by Asymptotic Methods

通过渐近方法加速封闭系统的性能推理

• DOI: 10.1145/3143314.3078514
• 发表时间: 2017
• 期刊: ACM SIGMETRICS Performance Evaluation Review
• 作者: Casale G

Understanding, modelling, and improving the performance of web applications in multicore virtualised environments

• DOI: 10.1145/2568088.2568102
• 发表时间: 2014-03
• 期刊: Proceedings of the 5th ACM/SPEC international conference on Performance engineering
• 作者: Xi Chen;C. Ho;Rasha Osman;P. Harrison;W. Knottenbelt