Doctoral Dissertation Research in DRMS: Developing and Validating a Method of Coherence-Based Judgment Aggregation

DRMS 博士论文研究：开发和验证基于一致性的判断聚合方法

• 批准号: 1919055
• 负责人: David Budescu
• 金额: $ 1.99万
• 依托单位: Fordham University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2021-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1919055&HistoricalAwards=false
• 关键词: Doctoral; Dissertation; Research; DRMS; Developing

High quality forecasting is essential to any area where people make decisions based on uncertain information, such as in geopolitics, the environment, and medical and financial domains. Subjective judgements of probabilities that quantify uncertainty are often used to produce such predictions. Previous research has differentiated two qualities of forecasts: accuracy, and coherence, or the extent of logical and probabilistic consistency in judgments. Recent work has suggested that coherent judgments tend to also be more accurate. Yet there exists no measure of coherence that might be used to identify a "select crowd" of judges who are more accurate than the average. The purpose of this research is to develop a psychological measure of coherence and determine how it predicts accuracy. Specifically, the research will compare the relative accuracy of judgments that are weighed by this new measure with judgments weighed by existing methods of judgment aggregation. Results of this work can help improve judgment quality in highly consequential fields. To answer this question, the researchers will create items that measure various aspects of probabilistic coherence. For example, items would test whether the probabilities assigned to all possible outcomes add to 1, as expected. Researchers will assess the psychometric properties - reliability, convergent and divergent validity - of the newly devised measure. To test the hypothesis that individuals with higher coherence scores are also more accurate, the researchers will weigh judgments from an incentivized forecasting tournament with this newly devised coherence measure. They will compare the accuracy of judgments weighed by this new method with other existing methods, such as linear and logarithmic averaging. The coherence measure may also produce a new, out-of-sample way of identifying the experts without additional knowledge of forecast performance, which will further our understanding of the role that individual-level coherence plays in maximizing judgment quality.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

高质量的预测对于人们根据不确定信息做出决策的任何领域都至关重要，例如地缘政治、环境、医疗和金融领域。量化不确定性的概率的主观判断通常用于产生此类预测。先前的研究区分了预测的两个品质：准确性和连贯性，或者判断中逻辑和概率一致性的程度。最近的研究表明，连贯的判断往往也更加准确。然而，不存在可用于识别比平均水平更准确的“精选群体”法官的一致性衡量标准。这项研究的目的是开发一种连贯性的心理测量方法，并确定它如何预测准确性。 具体来说，该研究将比较用这种新衡量标准衡量的判断的相对准确性与用现有判断聚合方法衡量的判断的相对准确性。这项工作的结果可以帮助提高高度重要领域的判断质量。为了回答这个问题，研究人员将创建测量概率一致性各个方面的项目。 例如，项目将测试分配给所有可能结果的概率是否按预期加起来为 1。研究人员将评估新设计的测量方法的心理测量特性——可靠性、收敛效度和发散效度。 为了检验一致性分数较高的个体也更准确的假设，研究人员将使用这种新设计的一致性度量来权衡激励预测锦标赛的判断。他们将把这种新方法与其他现有方法（例如线性平均和对数平均）进行权衡的判断准确性进行比较。一致性度量还可能产生一种新的、样本外的方法来识别专家，而无需额外了解预测性能，这将进一步我们了解个人层面的一致性在最大化判断质量方面所发挥的作用。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。