CSR: Small: Empower Data-Intensive Computing: the integrated data management approach

CSR：小：赋能数据密集型计算：集成数据管理方法

• 批准号: 1526887
• 负责人: Xian-He Sun
• 金额: $ 40万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1526887&HistoricalAwards=false
• 关键词: CSR; Small; Empower; Data; Intensive

From the computer system point of view there are two types of digital data: observational data, the data collected by electrical devices such as sensor, monitor, camera, text, etc.; and simulation data, data generated by computing. The former represents newly emerged internet data-driven applications, such as social media and data analytic; and the latter represents the conventional computing-driven applications, such as climate modeling and computational fluid dynamics. In general, the latter requires strong consistency for correctness and the former does not. The difference in consistency leads to two kinds of file systems: data-intensive distributed file system, represented by the MapReduce-based Hadoop distributed file systems (HDFS); and computing-intensive file systems, represented by the high performance parallel file systems (PFS), such as the IBM general parallel file system (GPFS). These two kinds of file systems are designed with different philosophies, for different applications, and do not talk to each other. Understanding huge amounts of collected data depends on powerful computation, whereas large-scale computation requires the management of large data. Therefore, big data applications demand an integrated solution. The integrated data access system (IDAS) developed under this research is designed to bridge the data management gap.In agreement with the CAP theory in the distributed system design, the IDAS approach is not designed as a new standalone system but as a software layer which provides an integrated interface to conduct cross-platform data access, from HDFS to PFS, or from PFS to HDFS, read or write, effectively and interchangeably without changing the users' applications. The development plan for IDAS has three components: 1) establish the communication channels so that data can be accessed between HDFS and PFS; 2) design an extended semantic interface so that different file systems can be accessed under different computing systems; 3) develop optimization techniques to optimize I/O operation under HDFS, PFS, and under IDAS. Big data requires a joint effort of the data-driven internet computing community and the compute-driven scientific computing community. IDAS provides a sustainable, cost-effective infrastructure for cross-platform, cross-community services of data storage, access, and sharing. This research will create advanced solutions and technologies that will have direct impact on improving the efficiency of data access and management at scale. Since big data is a national strategic infrastructure for science, engineering, and industry, the proposed investigations will advance a broad range of fields. The success of this research will strive to make significant progress of a timely, important, highly challenging, and high-impact problem, namely integrated data access system.

从计算机系统的角度来看，有两种类型的数字数据：观察数据，传感器，显示器，相机，文本等电气设备收集的数据；和仿真数据，通过计算生成的数据。前者代表了新出现的Internet数据驱动的应用程序，例如社交媒体和数据分析；后者代表了传统的计算驱动的应用，例如气候建模和计算流体动力学。通常，后者需要强大的正确性，而前者则不需要。一致性的差异导致两种文件系统：基于MapReduce的Hadoop分布式文件系统（HDFS）表示数据密集的分布式文件系统；以及以高性能并行文件系统（PFS）为代表的计算密集型文件系统，例如IBM常规并行文件系统（GPFS）。这两种文件系统的设计具有不同的哲学，用于不同的应用程序，并且不相互交流。了解大量收集的数据取决于强大的计算，而大规模计算需要大量数据的管理。因此，大数据应用程序需要集成解决方案。根据本研究开发的集成数据访问系统（IDA）旨在与分布式系统设计中的CAP理论一致弥合数据管理差距，IDAS方法不是作为新的独立系统设计设计的，而是作为一个软件层，该软件层提供了一个集成的界面，以进行跨平台数据访问，从HDF到PFS到PFS，或者从PFS到HDFS，不用hdfs sepriptions offersive offly offers，又有效率地，并且有效地将其互动，并有效地互动，并互动，并互动。 IDA的开发计划具有三个组成部分：1）建立通信渠道，以便可以在HDFS和PFS之间访问数据； 2）设计扩展的语义界面，以便可以在不同的计算系统下访问不同的文件系统； 3）开发优化技术，以优化HDF，PFS和IDA下的I/O操作。大数据需要数据驱动的互联网计算社区和计算驱动的科学计算社区的共同努力。 IDA为数据存储，访问和共享的跨平台，跨社区服务提供了可持续的，具有成本效益的基础架构。这项研究将创建高级解决方案和技术，这些解决方案和技术将直接影响提高数据访问和管理的效率。由于大数据是科学，工程和行业的国家战略基础设施，因此拟议的调查将推进广泛的领域。这项研究的成功将努力在及时，重要，高度挑战和高影响力问题（即集成数据访问系统）方面取得重大进展。