CAREER: Advancing Open-Ended Crowdsourcing: The Next Frontier in Crowdsourced Data Management

职业：推进开放式众包：众包数据管理的下一个前沿

• 批准号: 1940757
• 负责人: Aditya Parameswaran
• 金额: $ 41.34万
• 依托单位: University of California-Berkeley
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-05-15 至 2024-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1940757&HistoricalAwards=false
• 关键词: CAREER; Advancing; Open; Ended; Crowdsourcing

Machine learning on big data is finally having an impact on our daily lives, from small triumphs like Siri and Google Translate to much tougher emerging applications like driverless cars and computer-assisted medical image diagnosis. From mundane online fraud detection to the most sophisticated uses of computer vision, these applications share an insatiable appetite for massive labeled training data. The primary source of high-quality labels is crowdsourcing, and research to date on crowdsourcing has focused on the key problem of how to maximize the production of high-quality crowdsourced labels per dollar spent, for problems where workers must choose between just a few predefined labels. However, more open-ended labeling problems have grown to constitute almost half of crowdsourced tasks today, and open-ended tasks raise an entirely new set of research challenges for crowdsourced data management.This activity addresses the key new research challenges in managing and optimizing open-ended crowdsourcing. Since open-ended crowdsourcing employs tasks with a large number of alternatives, humans struggle to select error-free ones. Additional challenges emerge in determining the open-ended task types appropriate for a specific problem, developing schemes to ascertain the right answer given open-ended worker responses, and inferring the hidden perspectives behind worker answers. The activity targets open-ended crowdsourcing problems that span nearly 90% of those used in practice today, with wide applicability in computer vision, natural language processing, and machine learning in general. The technical outcomes of the activity include the first foundational principles for open-ended crowdsourced data management, which in turn will expand the reach of machine learning into new and more challenging domains and more effective solutions in existing applications that impact our everyday lives. The pedagogical outcomes of the activity include a course on human-in-the-loop data analytics, crowdsourcing education modules for school teachers, as well as a quantification and dissemination of how crowdsourcing is performed in practice, along with a benchmark to accelerate crowdsourcing research in the future.

大数据上的机器学习终于对我们的日常生活产生了影响，从 Siri 和谷歌翻译这样的小胜利，到无人驾驶汽车和计算机辅助医学图像诊断等更严峻的新兴应用。 从普通的在线欺诈检测到最复杂的计算机视觉应用，这些应用程序都对大量标记的训练数据有着永不满足的胃口。高质量标签的主要来源是众包，迄今为止，关于众包的研究主要集中在如何最大限度地提高每一美元生产的高质量众包标签的关键问题，以解决工人必须在几个预定义的标签之间进行选择的问题。标签。 然而，更多的开放式标签问题已发展成为当今众包任务的近一半，开放式任务为众包数据管理提出了一系列全新的研究挑战。这项活动解决了管理和优化开放式任务中的关键新研究挑战。结束众包。由于开放式众包采用具有大量替代方案的任务，因此人类很难选择无错误的任务。在确定适合特定问题的开放式任务类型、制定方案以确定给定开放式工作人员响应的正确答案以及推断工作人员答案背后隐藏的观点方面，还出现了其他挑战。该活动针对开放式众包问题，涵盖当今实践中近 90% 的问题，在计算机视觉、自然语言处理和机器学习中具有广泛的适用性。该活动的技术成果包括开放式众包数据管理的首要基本原则，这反过来又会将机器学习的范围扩大到新的、更具挑战性的领域，并在影响我们日常生活的现有应用程序中提供更有效的解决方案。该活动的教学成果包括人机循环数据分析课程、学校教师的众包教育模块、众包在实践中如何进行的量化和传播，以及加速众包研究的基准将来。

项目成果

An Exploratory User Study of Visual Causality Analysis

视觉因果分析的探索性用户研究

• DOI: 10.1111/cgf.13680
• 发表时间: 2019-06
• 期刊: Computer Graphics Forum
• 影响因子: 2.5
• 作者: Yen, Chi‐Hsien Eric;Parameswaran, Aditya;Fu, Wai‐Tat
• 通讯作者: Fu, Wai‐Tat

Flexible rule-based decomposition and metadata independence in modin: a parallel dataframe system

modin 中灵活的基于规则的分解和元数据独立性：并行数据框架系统

• DOI: 10.14778/3494124.3494152
• 发表时间: 2021-11
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Petersohn, Devin;Tang, Dixin;Durrani, Rehan;Melik;Gonzalez, Joseph E.;Joseph, Anthony D.;Parameswaran, Aditya G.
• 通讯作者: Parameswaran, Aditya G.

Fine-Grained Lineage for Safer Notebook Interactions.

细粒度的血统可实现更安全的笔记本交互。

• DOI: 10.14778/3447689.3447712
• 发表时间: 2021-02
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Stephen Macke; Hongpu Gong
• 通讯作者: Hongpu Gong

Lux: always-on visualization recommendations for exploratory dataframe workflows

Lux：针对探索性数据框工作流程的始终在线可视化建议

• DOI: 10.14778/3494124.3494151
• 发表时间: 2021-11
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Lee, Doris Jung;Tang, Dixin;Agarwal, Kunal;Boonmark, Thyne;Chen, Caitlyn;Kang, Jake;Mukhopadhyay, Ujjaini;Song, Jerry;Yong, Micah;Hearst, Marti A.;et al
• 通讯作者: et al

From Sketching to Natural Language: Expressive Visual Querying for Accelerating Insight

从草图到自然语言：用于加速洞察力的富有表现力的视觉查询

• 发表时间: 2021-01
• 期刊: SIGMOD record
• 影响因子: 1.1
• 作者: Siddiqui, T;Luh P;Wang Z;Karahalios K;Parameswaran A
• 通讯作者: Parameswaran A