Collaborative Research: RI: Medium: Learning Joint Crowd-Space Embeddings for Cross-Modal Crowd Behavior Prediction

合作研究：RI：Medium：学习联合人群空间嵌入以进行跨模式人群行为预测

• 批准号: 1955365
• 负责人: Sejong Yoon
• 金额: $ 16.25万
• 依托单位: The College of New Jersey
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1955365&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; Learning

Many societal activities, including air transport, disaster remediation, social events such as concerts and sports, require efficient and effective methodologies for monitoring, understanding, and reacting to behaviors of large concentrations of people, the crowds, that give rise to those events. Simultaneously, the type and evolution of those behaviors are intimately tied to the form and function of the environments where they occur. As crowds increase in size or change their actions in response to intrinsic or extrinsic factors, it is critical for the built environments, including their future designs, to adapt to those changes. Present-day technological tools aim to analyze and predict the link between crowds and environments. However, they rely on rigid, hand-tuned, computationally costly simulation models, severely limiting their practical utility. This project seeks to bridge this gap by devising a novel way of modeling the inherent relationship between the structure and semantics of complex environments, and the presence and behavior of its human occupants, from small groups to dense crowds. The main goal is to predict crowd behavior accurately, from microscopic motion to aggregate crowd dynamics, in novel, never-before-seen environment configurations using Neuro-Cognitive Modeling of Environments and Humans (NUCLEUM) to replace the computationally expensive yet often mismatched-with-reality physical simulations. To accomplish this goal, this project collaboratively seeks to tackle the problem of predicting crowd behavior in complex environments by learning data-driven models that will seamlessly "translate" between different representations of crowds and their environments. Specifically, this project has three main research thrusts: (Thrust 1) Learning a Joint Crowd-Space Representation. The project will develop a novel multi-concept transfer learning framework to enable coupled learning across three highly heterogeneous concepts: (a) environment layouts (e.g., floor plans), (b) macroscopic crowd properties (e.g., flow), and (c) microscopic crowd trajectories. Once learned, the framework will enable predictions of flow patterns of a crowd, directly from the layout of an environment and vice versa. (Thrust 2) A Hybrid Multi-modal Corpus of Environment Contexts and Crowd Movement. This project will create a novel hybrid multi-modal corpus of environmental contexts and crowd behavior, which will leverage data from field observations, controlled laboratory experiments, crowd simulations, and multi-user virtual reality platforms. This corpus will allow training models that generalize across the space of environment and crowd conditions. (Thrust 3) Model Evaluation, Applications, and Use Cases. Trained models' robustness will be evaluated in terms of their ability to produce valid crowd trajectories, which are statistically similar to ground truth observations while generalizing to the new, unseen crowd, and environmental contexts. This project will subsequently apply the trained models in a variety of application contexts on real-world built and yet-to-be-built environments to predict crowd behavior in unseen environments, identify vulnerabilities in environments, and reconfigure environment designs to improve crowd behavior.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

许多社会活动，包括航空运输，灾难修复，音乐会和体育等社会活动，都需要高效有效的方法来监测，理解和对大量人群（人群）的行为产生这些事件的行为。同时，这些行为的类型和演变与发生环境的形式和功能密切相关。随着人群的规模增加或因响应内在或外在因素而改变其行动，对于建筑环境（包括未来的设计）来说，适应这些变化至关重要。当今的技术工具旨在分析和预测人群与环境之间的联系。但是，他们依靠刚性，手工调整，计算上昂贵的仿真模型，严重限制了其实际实用性。该项目旨在通过设计一种新颖的方式来建模复杂环境的结构和语义之间的固有关系，以及其人类居住者的存在和行为，从小组到密集的人群。主要目标是在新颖的，从未见过的环境配置中，使用环境和人类的神经认知建模（核）来替换计算上昂贵但经常不匹配的与 - 与现实的物理模拟，从未见过的环境建模，从微观的运动到汇总人群动态，从微观运动到汇总人群动态，从而准确地预测了人群的行为。为了实现这一目标，该项目旨在通过学习数据驱动的模型来解决复杂环境中人群行为的问题，这些模型将在人群及其环境的不同表示之间无缝地“翻译”。具体来说，该项目具有三个主要的研究作用：（推力1）学习联合人群代表。该项目将开发一个新颖的多概念转移学习框架，以使三个高度异构概念的耦合学习能够：（a）环境布局（例如，平面图），（b）宏观人群属性（例如，流动）和（c）微观人群轨迹。一旦获悉，该框架将直接从环境的布局直接实现人群流动模式的预测，反之亦然。 （推力2）混合多模式的环境环境和人群运动。该项目将创建一个新型的混合多模式环境环境和人群行为的语料库，该语料将利用现场观察，受控实验室实验，人群模拟和多用户虚拟现实平台的数据来利用数据。该语料库将允许培训模型在整个环境和人群条件的范围内推广。 （推力3）模型评估，应用和用例。训练有素的模型的鲁棒性将根据其产生有效的人群轨迹的能力进行评估，这些轨迹在统计学上与地面真理观察相似，同时将其推广到新的，看不见的人群和环境环境。该项目随后将在各种申请环境中应用训练有素的模型，以对现实且尚未建立的环境进行预测，以预测看不见的环境中的人群行为，确定环境中的脆弱性，并重新配置环境设计以改善人群的行为。该奖项反映了NSF的法定任务，并通过评估基金会的范围来反映出支持者的支持者，并通过基金会的范围进行了评估和宽广的影响。

项目成果

MSI: Maximize Support-Set Information for Few-Shot Segmentation

• DOI: 10.1109/iccv51070.2023.01765
• 发表时间: 2022-12
• 期刊: 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Seonghyeon Moon;Samuel S. Sohn;Honglu Zhou;Sejong Yoon;V. Pavlovic;Muhammad Haris Khan;M. Kapadia

HOPPER: MULTI-HOP TRANSFORMER FOR SPATIOTEMPORAL REASONING

HOPPER：用于时空推理的多跳变压器

• 发表时间: 2021
• 期刊: International Conference on Learning Representation (ICLR
• 作者: Zhou, H.;Kadav, A.;Lai, F.;Niculescu-Mizil, A.;Min, M.R.;Kapadia, M.;Graf, H.P.
• 通讯作者: Graf, H.P.

Constructivist Approaches for Computational Emotions: A Systematic Survey

计算情感的建构主义方法：系统调查

• 发表时间: 2023
• 期刊: Computational Theory of Mind for Human-Machine Teams. AAAI-FSS 2021. Lecture Notes in Computer Science
• 作者: Viola, Alexander;Pavlovic, Vladimir;Yoon, Sejong
• 通讯作者: Yoon, Sejong

A2X: An end-to-end framework for assessing agent and environment interactions in multimodal human trajectory prediction

A2X：用于评估多模式人类轨迹预测中代理和环境交互的端到端框架

• DOI: 10.1016/j.cag.2022.05.010
• 发表时间: 2022
• 期刊: Computers & Graphics
• 作者: Sohn, Samuel S.;Lee, Mihee;Moon, Seonghyeon;Qiao, Gang;Usman, Muhammad;Yoon, Sejong;Pavlovic, Vladimir;Kapadia, Mubbasir
• 通讯作者: Kapadia, Mubbasir

Graph-based generative representation learning of semantically and behaviorally augmented floorplans

• DOI: 10.1007/s00371-021-02155-w
• 发表时间: 2020-12
• 期刊: The Visual Computer
• 作者: Vahid Azizi;Muhammad Usman;H. Zhou;P. Faloutsos;M. Kapadia