RI: Small: Decision-Theoretic Control of Crowd-Sourced Workflows

RI：小型：众包工作流程的决策理论控制

• 批准号: 1016713
• 负责人: Daniel Weld
• 金额: $ 30.47万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-09-15 至 2014-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1016713&HistoricalAwards=false
• 关键词: RI; Small; Decision; Theoretic; Control

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people as an open request for services. Requesters use crowd-sourcing for a wide variety of jobs like dictation-transcription, content screening, linguistic tasks, user-studies, etc. These requesters often use complex workflows to subdivide a large task into bite-sized pieces (including the management of these tasks), each of which is independently crowd-sourced. These workflows are paramount to the success of crowd-sourcing, still, there has been little attention paid to methods for dynamically optimizing the throughput of a workflow. Controlling and optimizing such a workflow is an excellent application for AI research for two reasons. First, it is challenging in that the agent has to understand the dynamics of an uncertain, real-time environment and reason about distinct choices for a decision. More importantly, the domain has significant economic value -- progress can potentially impact hundreds of thousands of people and spur economic development in a fast growing sector.This project is investigating complex workflows using a decision-theoretic framework that optimizes for a quality/price trade-off, with aims of (1) building statistical models of worker behavior derived from a large corpus of online behavior, (2) defining a declarative representation language to describe a wide range of workflows, and (3) developing an automated scheme that optimizes a general workflow resulting in an automated controller for making informed decisions at various stages of the process and for monitoring worker accuracies and computing corrections based on them. In the longer term, perhaps beyond the scope of this project, is (4) development of an interface optimizer that automatically learns the best user interface for a task based on user behavior increasing throughput of the workflow, and (5) integrating these ideas in an open-source, software toolkit to directly benefit the various requesters in managing their tasks.

众包是一种最新的框架，其中将人类智能任务作为公开的服务请求外包给一群不知名的人。请求者将众包用于各种各样的工作，例如听写转录、内容筛选、语言任务、用户研究等。这些请求者经常使用复杂的工作流程将大型任务细分为小块（包括这些任务的管理）。任务），每个任务都是独立众包的。这些工作流程对于众包的成功至关重要，但动态优化工作流程吞吐量的方法却很少受到关注。控制和优化这样的工作流程对于人工智能研究来说是一个很好的应用，原因有两个。首先，它具有挑战性，因为代理必须了解不确定的实时环境的动态，并推理不同的决策选择。更重要的是，该领域具有重大的经济价值——进步可能会影响数十万人，并刺激快速增长行业的经济发展。该项目正在使用决策理论框架来研究复杂的工作流程，该框架可优化质量/价格交易-off，目标是 (1) 构建源自大量在线行为的工作人员行为统计模型，(2) 定义声明性表示语言来描述广泛的工作流程，以及 (3) 开发优化的自动化方案产生自动化控制器的一般工作流程用于在流程的各个阶段做出明智的决策，并监控工人的准确性并据此计算修正。从长远来看，也许超出了本项目的范围，（4）开发一个界面优化器，根据用户行为自动学习任务的最佳用户界面，增加工作流程的吞吐量，以及（5）将这些想法集成到一个开源软件工具包，可直接使各种请求者管理其任务受益。