CSR: Small: Scalable Fine-Grain Lineage for Debugging Data-Intensive Workflows

CSR：小型：用于调试数据密集型工作流程的可扩展细粒度谱系

• 批准号: 1219220
• 负责人: Alin Deutsch
• 金额: $ 45万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-01 至 2018-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1219220&HistoricalAwards=false
• 关键词: CSR; Small; Scalable; Fine; Grain

This project addresses the next major impediment to the continued adoptionof "big-data" analytics---the management of their life cycle, whichincludes debugging, tuning, and auditing. Today, data-intensive analyticsare improving operations across multiple industries, translating terabytesof raw data into useful data analysis. Taking advantage of big data willbe necessary to sustain competitive advantages for areas ranging from powergeneration, to retail, oil exploration, manufacturing, various scientificdisciplines, and national security. However, the extreme scalability ofthese data processing architectures hides inefficiencies and obfuscatesperformance analysis, creating both obvious and hidden costs to theiradoption. Tuning and debugging large data-intensive workflows is currentlya black art that mostly consists of tedious manual analysis.The research seeks to dramatically alter how data scientists design anddebug their analytics to sidestep this authoring and deployment bottleneck.In particular, the PI's are developing scalable, efficient architecturesfor capturing fine-grain data lineage, information that tracks the use ofdata through the analytic pipeline, from a range of data-intensive scalablecomputing (DISC) systems. Such lineage serves as a basis for discoveringinefficiencies and suggesting optimizations via step-wise debugging, faulttracing, anomaly detection, and lineage-driven data cleaning and datamining. The development and open-source release of such lineage-captureand analysis platforms promises to dramatically accelerate the adoption ofbig-data analytics.

该项目解决了继续采用“大数据”分析的下一个主要障碍——生命周期的管理，包括调试、调整和审计。 如今，数据密集型分析正在改善多个行业的运营，将数 TB 的原始数据转化为有用的数据分析。 利用大数据对于维持从发电、零售、石油勘探、制造、各种科学学科和国家安全等领域的竞争优势是必要的。 然而，这些数据处理架构的极端可扩展性隐藏着效率低下和性能分析混乱的问题，从而为它们的采用带来了明显和隐性的成本。 调整和调试大型数据密集型工作流程目前是一门黑术，主要由繁琐的手动分析组成。该研究旨在极大地改变数据科学家设计和调试其分析的方式，以避开这种创作和部署瓶颈。特别是，PI 正在开发可扩展的、用于从一系列数据密集型可扩展计算 (DISC) 系统中捕获细粒度数据沿袭的高效架构，即通过分析管道跟踪数据使用情况的信息。 这种沿袭可以作为发现效率低下的基础，并通过逐步调试、故障跟踪、异常检测以及沿袭驱动的数据清理和数据挖掘提出优化建议。 这种谱系捕获和分析平台的开发和开源发布有望大大加速大数据分析的采用。