Multimodal Visitor Analytics: Investigating Naturalistic Engagement with Interactive Tabletop Science Exhibits

多模式访客分析：研究交互式桌面科学展览的自然参与

• 批准号: 1713545
• 负责人: James Lester
• 金额: $ 195.2万
• 依托单位: North Carolina State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-03-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1713545&HistoricalAwards=false
• 关键词: Multimodal; Visitor; Analytics; Investigating; Naturalistic

Engagement is the cornerstone of learning in informal science education. During free-choice learning in museums and science centers, visitor engagement shapes how learners interact with exhibits, navigate through exhibit spaces, and form attitudes, interests, and understanding of science. Recent advances in multimodal learning analytics are creating novel opportunities for expanding the range and richness of measures of visitor engagement in free-choice settings. In particular, multimodal learning analytics offer significant potential for integrating multiple data sources to devise a composite picture of visitors' cognitive, affective, and behavioral engagement. The project will center on providing a rich empirical account of meaningful visitor engagement with interactive tabletop science exhibits among individual visitors and small groups, as well as uncovering broader tidal patterns in visitor engagement that unfold across exhibit spaces. A key objective of the project is creating models and practitioner-focused learning analytic tools that will inform the best practices of exhibit designers and museum educators. This project is funded by the Advancing Informal STEM Learning (AISL) program. As part of its overall strategy to enhance learning in informal environments, AISL funds research and innovative approaches and resources for use in a variety of settings.The research team will conduct data-rich investigations of visitors' learning experiences with multimodal learning analytics that fuse the rich multichannel data streams produced by fully-instrumented exhibit spaces with the data-driven modeling functionalities afforded by recent advances in machine learning and educational data mining. The research team will conduct a series of visitor studies of naturalistic engagement in solo, dyad, and group interactions as visitors explore interactive tabletop science exhibits. The studies will utilize eye trackers to capture visitors' moment-to-moment attention, facial expression analysis and quantitative field observations to track visitors' emotional states, trace logs generated by exhibit software, as well as motion-tracking sensors and coded video recordings to capture visitors' behavioral interactions. The studies will also use conversation recordings and pre-post assessment measures to capture visitors' science understanding and inquiry processes. With these multimodal data streams as training data, the research team will use probabilistic and neural machine learning techniques to devise learning analytic models of visitor engagement. The project will be conducted by a partnership between North Carolina State University and the North Carolina Museum of Natural Sciences. The research team will 1) design a data-rich multimodal visitor study methodology, 2) create the Visitor Informatics Platform, a suite of open source software tools for multimodal visitor analytics, and 3) launch the Multimodal Visitor Data Warehouse, a curated visitor experience data archive. Together, the multimodal visitor study methodology, the Visitor Informatics Platform, and the Multimodal Visitor Data Warehouse will enable researchers and practitioners in the informal science education community to utilize multimodal learning analytics in their own informal learning environments. It is anticipated that the project will advance the field of informal STEM learning by extending and enriching measures of meaningful visitor engagement, expanding the evidence base for visitor experience design principles, and providing learning analytic tools to support museum educators. By enhancing understanding of the cognitive, affective, and behavioral dynamics underlying visitor experiences in science museums, informal science educators will be well-positioned to design learning experiences that are more effective and engaging.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

参与是非正式科学教育学习的基石。在博物馆和科学中心的自由选择学习期间，游客参与塑造了学习者如何与展览互动，浏览展览空间，并形成态度，兴趣和对科学的理解。多模式学习分析的最新进展正在创造新的机会，以扩大自由选择环境中访客参与度的范围和丰富性。尤其是，多模式学习分析为整合多个数据源提供了重要的潜力，以设计访客认知，情感和行为参与的综合图片。该项目将集中在提供有意义的访客参与度与互动式桌面科学展览中的丰富经验描述，并在各个访问者和小组之间展出，并揭示了在展览空间中展开的访客参与中更广泛的潮汐模式。该项目的一个关键目标是创建模型和以从业者为中心的学习分析工具，这些工具将为展览设计师和博物馆教育工作者的最佳实践提供信息。该项目由前进的非正式STEM学习（AISL）计划提供资金。 作为增强非正式环境中学习的整体策略的一部分，AISL资助了在各种环境中使用的研究和创新方法和资源。研究团队将通过融合多模式学习分析的访问者学习经验的数据丰富的研究，这些学习经验融合了由富裕的多渠道数据流提供的富裕的多渠道数据流，这些数据通过全面启动的数据进行了用于数据的空间，该数据通过可负担得出的功能进行了构图，以构建功能范围内的范围内的功能范围内的范围。研究团队将在访问者探索互动式桌面科学展览会时进行一系列关于自然主义参与的游客研究。这些研究将利用眼动追踪器来捕捉访客的瞬间关注，面部表达分析和定量现场观察，以跟踪访客的情绪状态，展览软件产生的痕量日志以及运动跟踪传感器和编码视频记录以捕捉访客的行为互动。这些研究还将使用对话记录和验证前评估措施来捕捉访客的科学理解和询问过程。以这些多模式数据流作为培训数据，研究团队将使用概率和神经机器学习技术来设计访客参与的学习分析模型。该项目将由北卡罗来纳州立大学和北卡罗来纳州自然科学博物馆之间的合作关系进行。研究团队将1）设计一种数据丰富的多模式访问者研究方法，2）创建访客信息平台，用于多模式访问者分析的开源软件工具套件，以及3）启动多模式访问者数据仓库，策划的访问者体验数据档案。多模式访问者研究方法，访问者信息学平台和多模式访问者数据仓库将使非正式科学教育社区的研究人员和从业人员能够在其自己的非正式学习环境中利用多模式学习分析。可以预计，该项目将通过扩展和丰富有意义的访客参与度，扩大访客体验设计原理的证据基础并提供学习分析工具来支持博物馆教育工作者的证据基础，通过扩展有意义的访客参与度，扩大有意义的访客参与度，扩大有意义的访客参与度的衡量标准，来推进非正式的STEM学习领域。通过增强对科学博物馆访客经历的认知，情感和行为动态的理解，非正式的科学教育工作者将非常有用，以设计更有效且引人入胜的学习经验。该奖项反映了NSF的法定任务，并通过使用该基金会的知识分子优点和广泛的影响来评估NSF的法定任务，并被视为值得的支持。

项目成果

What's Fair is Fair: Detecting and Mitigating Encoded Bias in Multimodal Models of Museum Visitor Attention

• DOI: 10.1145/3462244.3479943
• 发表时间: 2021-10
• 期刊: Proceedings of the 2021 International Conference on Multimodal Interaction
• 作者: Halim Acosta;Nathan L. Henderson;Jonathan P. Rowe;Wookhee Min;James Minogue;James C. Lester

Multimodal Trajectory Analysis of Visitor Engagement with Interactive Science Museum Exhibits

游客与互动科学博物馆展品互动的多模态轨迹分析

• DOI: 10.1007/978-3-030-78270-2_27
• 发表时间: 2021
• 期刊: Artificial Intelligence in Education
• 作者: Emmerson, Andrew;Henderson, Nathan;Min, Wookhee;Rowe, Jonathan;Minogue, James;Lester, James
• 通讯作者: Lester, James

Early Prediction of Visitor Engagement in Science Museums with Multimodal Learning Analytics

• DOI: 10.1145/3382507.3418890
• 发表时间: 2020-10-21
• 期刊: ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction
• 作者: Emerson, Andrew;Henderson, Nathan;Lester, James
• 通讯作者: Lester, James

Early Prediction of Visitor Engagement in Science Museums with Multimodal Adversarial Domain Adaptation

通过多模态对抗域适应对科学博物馆的游客参与度进行早期预测

• 发表时间: 2021
• 期刊: Educational Data Mining
• 作者: Henderson, Nathan;Min, Wookhee;Emerson, Andrew;Rowe, Jonathan;Lee, Seung;Minogue, James;Lester, James
• 通讯作者: Lester, James

Investigating Visitor Engagement in Interactive Science Museum Exhibits with Multimodal Bayesian Hierarchical Models

使用多模态贝叶斯分层模型调查游客对互动科学博物馆展览的参与度

• DOI: 10.1007/978-3-030-52237-7-14
• 发表时间: 2020
• 期刊: Proceedings of the Twenty-First International Conference on Artificial Intelligence in Education
• 作者: Emerson, Andrew;Henderson, Nathan;Rowe, Jonathan;Min, Wookhee;Lee, Seung;Minogu, James;Lester, James.
• 通讯作者: Lester, James.