III: Medium: Collaborative Research: DataHub - A Collaborative Dataset Management Platform for Data Science

III：媒介：协作研究：DataHub - 数据科学协作数据集管理平台

• 批准号: 1513407
• 负责人: Aditya Parameswaran
• 金额: $ 33.3万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2018-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1513407&HistoricalAwards=false
• 关键词: III; Medium; Collaborative; Research; DataHub

The rise of the Internet, smart phones, and wireless sensors has resulted in a vast trove of data about all aspects of our lives, from our social interactions to our personal preferences to our vital signs and medical records. Increasingly, "data science" teams want to collaboratively analyze these datasets, to understand trends and to extract actionable business, scientific, or social insights. Unfortunately, while there exist tools to support data analysis, much-needed underlying infrastructure and data management capabilities are missing. To this end, "DataHub", a collaborative platform for cleaning, storing, understanding, sharing, and publishing datasets, will be developed. DataHub will be a publicly accessible platform that will host private user datasets as well as public datasets retrieved from online sources. DataHub will serve as the common substrate for data science, freeing up end users from tedious dataset book-keeping tasks, and instead supporting them in their search for useful insights. DataHub will be deployed on a large scale at MIT; partnerships with organizations and groups from a variety of sectors will be leveraged upon to show benefits for real data scientists and to ensure that the proposed techniques meet real-world big data challenges. The curriculum development part of this project will lead to the training of new data scientists, and the project will also provide opportunities for graduate and undergraduate students to participate in research and learn how to do collaborative research.Unlike most systems that focus on improving performance or on supporting even more sophisticated analyses, DataHub will instead focus on simplifying and automating many fundamental book-keeping operations that are a pre-requisite to data science. Key features of DataHub will include: (1) a flexible, source code control-like versioning system for data, that efficiently branches, merges, and differences datasets; (2) new data ingest, cleaning, and wrangling tools designed to automate data cleaning process; (3) the ability to search for "related" tables and to integrate them into the analysis process; and (4) the ability to selectively share and collaborate on data sets across users and teams. Overall, DataHub will significantly reduce the amount of effort involved on the part of data scientists for preparing, analyzing, sharing, and managing data.For more information, see the project website at: http://data-hub.org

互联网、智能手机和无线传感器的兴起产生了有关我们生活各个方面的大量数据，从我们的社交互动到我们的个人偏好，再到我们的生命体征和医疗记录。 “数据科学”团队越来越希望协作分析这些数据集，了解趋势并提取可操作的商业、科学或社会见解。不幸的是，虽然存在支持数据分析的工具，但缺少急需的底层基础设施和数据管理功能。为此，将开发一个用于清理、存储、理解、共享和发布数据集的协作平台“DataHub”。 DataHub 将是一个可公开访问的平台，将托管私人用户数据集以及从在线资源检索的公共数据集。 DataHub 将作为数据科学的通用基础，将最终用户从繁琐的数据集簿记任务中解放出来，并支持他们寻找有用的见解。 DataHub将在MIT大规模部署；将利用与各个部门的组织和团体的伙伴关系来展示真正的数据科学家的好处，并确保所提出的技术能够满足现实世界的大数据挑战。 该项目的课程开发部分将导致新数据科学家的培训，该项目还将为研究生和本科生提供参与研究并学习如何进行协作研究的机会。与大多数专注于提高性能或为了支持更复杂的分析，DataHub 将专注于简化和自动化许多基本的簿记操作，这是数据科学的先决条件。 DataHub 的主要功能包括：（1）一个灵活的、类似源代码控制的数据版本控制系统，可以有效地分支、合并和差异数据集； (2) 新的数据摄取、清理和整理工具，旨在自动化数据清理过程； (3) 搜索“相关”表格并将其整合到分析过程中的能力； (4) 跨用户和团队有选择地共享和协作数据集的能力。总体而言，DataHub 将显着减少数据科学家准备、分析、共享和管理数据所需的工作量。有关更多信息，请访问该项目网站：http://data-hub.org

项目成果

Optimally Leveraging Density and Locality for Exploratory Browsing and Sampling

最佳地利用密度和位置进行探索性浏览和采样

• DOI: 10.1145/3209900.3209903
• 发表时间: 2018-01
• 期刊: SIGMOD HILDA (Human-in-the-loop Data Analytics
• 作者: Kim, Albert;Xu, Liqi;Siddiqui, Tarique;Huang, Silu;Madden, Samuel;Parameswaran, Aditya
• 通讯作者: Parameswaran, Aditya