III: Small: RIOT: Statistical Computing with Efficient, Transparent I/O

III：小型：RIOT：具有高效、透明 I/O 的统计计算

• 批准号: 0916027
• 负责人: Jun Yang
• 金额: $ 50万
• 依托单位: Duke University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-09-01 至 2014-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0916027&HistoricalAwards=false
• 关键词: III; Small; RIOT; Statistical; Computing

Recent technological advances enable collection of massive amounts ofdata in science, commerce, and society. These datasets bring uscloser than ever before to solving important problems such as decodinghuman genomes and coping with climate changes. Meanwhile, theexponential growth in data volume creates an urgent challenge. Manyexisting analysis tools assume datasets fit in memory; when applied tomassive datasets, they become unacceptably slow because of excessivedisk input/output (I/O) operations.Across application domains, much of advanced data analysis is donewith custom programming by statisticians. Progress has been hinderedby the lack of easy-to-use statistical computing environments thatsupport I/O-efficient processing of large datasets. There have beenmany approaches toward I/O-efficiency, but none has gained tractionwith statisticians because of issues ranging from efficiency tousability. Disk-based storage engines and I/O-efficient functionlibraries are only a partial solution, because many sources ofI/O-inefficiency in programs remain at a higher, inter-operationlevel. Database systems seem to be a natural solution, with efficientI/O and a declarative language (SQL) enabling high-leveloptimizations. However, much work in integrating databases andstatistical computing remains database-centric, forcing statisticiansto learn unfamiliar languages and deal with their impedance mismatchwith host languages.To make a practical impact on statistical computing, this projectpostulates that a better approach is to make it transparent to usershow I/O-efficiency is achieved. Transparency means no SQL, or any newlanguage to learn. Transparency means that existing code should runwithout modification, and automatically gain I/O-efficiency. Theproject, nicknamed RIOT, aims at extending R---a widely popularopen-source statistical computing environment---to transparentlyprovide efficient I/O. Achieving transparency is challenging; RIOTdoes so with an end-to-end solution addressing issues on all fronts:I/O-efficient algorithms, pipelined execution, deferred evaluation,I/O-cost-driven expression optimization, smart storage andmaterialization, and seamless integration with an interpreted hostlanguage.RIOT integrates research and education, and continues the tradition ofinvolving undergraduates through REU and independent studies. As adatabase researcher, the PI is committed to learning and drawing fromwork from programming languages and high-performance computing.Findings from RIOT help create synergy and seed further collaborationwith these communities. To ensure practical impact on statisticalcomputing, RIOT has enlisted collaboration from statisticians and theR core development team on developing, evaluating, and disseminatingRIOT.Further information can be found at: http://www.cs.duke.edu/dbgroup/Main/RIOT

最新的技术进步使科学，商业和社会中的大量数量都可以收集。 这些数据集将USCLOSER比以往任何时候都带到解决重要问题，例如解码人类基因组和应对气候变化。 同时，数据量的指数增长带来了紧迫的挑战。 许多息息分析工具假设数据集适合内存；当应用Tomassive数据集时，由于过度风险输入/输出（I/O）操作，它们变得速度不高。 由于缺乏易于使用的统计计算环境，因此进展受到了阻碍。 有针对I/O效率的方法，但是由于效率可倾斜度等问题，没有人因统计学家而获得统计学家的关注。 基于磁盘的存储引擎和I/O高效功能元素只是一个部分解决方案，因为程序中的许多源/O-Inefliedy仍然处于较高的，跨式级别的范围。 数据库系统似乎是一种自然解决方案，具有高效/O和声明性语言（SQL）实现高级临时性化。 但是，在集成数据库和统计计算方面的许多工作仍然是以数据库为中心的，迫使统计学派学习不熟悉的语言并处理他们的阻抗不匹配的宿主语言。为了对统计计算产生实际影响，对统计计算的实际影响，这种项目可以更好地表明，一种更好的方法是使其能够使其透明化，以使其透明化I/O-o/o-o-o-feff effective。 透明度意味着没有SQL或任何新语言要学习。 透明度意味着现有代码应登录修改，并自动提高I/O效率。 绰号为Riot的项目旨在扩展R ---广泛的源源源统计计算环境，以透明地提供有效的I/O。 实现透明度是具有挑战性的； Riotdo借助端到端解决方案解决了所有方面的问题：I/O高效算法，管道执行，递延评估，I/O-COST驱动的表达优化，智能存储和智能存储和材料化以及与解释性的hostlanguage.Riot的无缝集成。Riot可以整合研究和教育，并继续进行逐步研究，并逐步进行逐步研究，并逐步进行了良好的努力和良好的效果。 作为Adatabase的研究人员，PI致力于从编程语言和高性能计算的工作中学习和绘制。暴动的发现有助于创造协同作用和种子进一步的合作，并在这些社区中进行进一步的协作。 为了确保对统计计算的实际影响，Riot已将统计学家和其他核心开发团队的合作吸引了开发，评估和散布。