Collaborative Research: Leveraging Crowd-AI Teams for Scalable Novelty Ratings of Heterogeneous Design Representations

协作研究：利用群体人工智能团队对异构设计表示进行可扩展的新颖性评级

• 批准号: 2231254
• 负责人: Faez Ahmed
• 金额: $ 30万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2026-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2231254&HistoricalAwards=false
• 关键词: Collaborative; Research; Leveraging; Crowd; AI

This research project will advance the field of engineering design by developing a framework for rating the novelty and quality of design artifacts using a combination of human and machine expertise. The project will address a significant challenge in the field, which is how to accurately evaluate the vast amounts of complex, heterogeneous design ideas generated in design studies. The project will contribute to our understanding of how to effectively leverage human and computational expertise in evaluating design artifacts, leading to improved product design. While focused on an engineering design use case, the project will contribute to machine learning (ML) research by providing evidence on what, when, and how to learn from humans when evaluating heterogeneous datasets. The project will establish a website with open-source data and workshops for early-career professionals to enhance the broader impacts of the work on the engineering design community at large. The research team will partner with the Women in Science and Engineering Research (WISER) program and the Multicultural Engineering Program (MEP) at Penn State to encourage participation of underrepresented groups in engineering.The project investigates three distinct means of evaluating heterogeneous design artifacts (CAD drawings, text, sketches, prototypes) against established design metrics at scale, namely (1) human expert designers and crowd-sourced human raters; (2) pure ML-based methods; and (3) expert designer-assisted ML methods. These methods will be used to better understand the impact of design representation on the reliability of crowd-based design ratings, establish and validate multi-modal representation learning methods for design evaluation, and create methods for deploying crowd-ML collaborations for scalable design evaluation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该研究项目将通过开发一个框架，结合人类和机器的专业知识来评估设计工件的新颖性和质量，从而推动工程设计领域的发展。该项目将解决该领域的一个重大挑战，即如何准确评估设计研究中产生的大量复杂、异构的设计理念。该项目将有助于我们了解如何有效地利用人力和计算专业知识来评估设计工件，从而改进产品设计。 该项目在专注于工程设计用例的同时，还将提供有关在评估异构数据集时向人类学习什么、何时以及如何学习的证据，从而为机器学习 (ML) 研究做出贡献。该项目将为早期职业专业人员建立一个包含开源数据和研讨会的网站，以增强该工作对整个工程设计界的更广泛影响。 该研究团队将与宾夕法尼亚州立大学的女性科学与工程研究 (WISER) 计划和多元文化工程计划 (MEP) 合作，鼓励代表性不足的群体参与工程。该项目研究了评估异构设计工件的三种不同方法（CAD图纸、文本、草图、原型）对照既定的大规模设计指标，即（1）人类专家设计师和众包人类评估者； (2) 纯基于机器学习的方法； (3) 专家设计者辅助的机器学习方法。 这些方法将用于更好地了解设计表示对基于人群的设计评级可靠性的影响，建立和验证用于设计评估的多模式表示学习方法，并创建部署人群机器学习协作以进行可扩展设计评估的方法。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。