OAC Core: Small: Open-Source Robust 4D Reconstruction Framework for Real-Time Dynamic Human Capture

OAC Core：小型：用于实时动态人体捕捉的开源稳健 4D 重建框架

• 批准号: 2007661
• 负责人: Xiaohu Guo
• 金额: $ 50万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007661&HistoricalAwards=false
• 关键词: OAC; Core; Small; Open; Source

The upcoming deployment of 5G technology makes it feasible to communicate with extremely low latency the vast amounts of data needed for Augmented Reality (AR) and Mixed Reality (MR), which enables transformative applications such as 3D tele-immersive (or 3D facetime) communication, mobile AR apps using captured 4D human contents, AI assistants of lifelike and personalized avatars, human-aware robots that can serve or work with humans, tele-rehabilitation to connect physical therapists with wounded patients far from treatment facilities, etc. All these examples of AR and MR applications require the development of real-time 4D (space and time) capture and reconstruction of dynamic scenes involving human bodies, faces, body add-ons (like clothes), and their surrounding environments. Despite prior research on real-time 4D reconstruction, there is still no open-source and robust reconstruction system that can model topological changes of the dynamic scenes and track the moving surfaces with accuracy and robustness. This project is designed to bridge such gaps, to develop an open-source and robust 4D reconstruction framework that can benefit researchers and developers in broader scientific communities as well as industry alliances, including 5G medical standards, AR and MR game engines, lifelike AI assistants, human-aware robotics, tele-rehabilitation, etc. This project also provides curriculum development and educational activities for graduate, undergraduate, and K-12 students through summer camps. The research proposed for this project centers around the elegant modeling of topological changes in dynamic scenes and the robust tracking of moving surfaces, towards the ultimate goal of developing an open-source robust 4D reconstruction framework for real-time capture of dynamic human scenes. To solve the challenges of topological changes, the volumetric fusion framework and its data structures will be fundamentally redesigned, by introducing Non-manifold Volumetric Grids into both Truncated Signed Distance Field (TSDF) and Embedded Deformation Graph (EDG) representations, allowing both the volumetric cells to replicate themselves and the edges to be broken. Such a novel topology-change-aware framework will allow the reconstructed mesh geometry to update its connectivity on-the-fly, along with a flexible deformation graph updating its connectivity between nodes throughout the 4D capture process. To solve the robust surface tracking problem in 4D dynamic human capture, a Parameterized Animatable Volumetric Model (PAVM) is proposed to combine the benefits of both the parametric human body model and the volumetric TSDF. The TSDF volumetric grids are built on top of the parameterized human body surfaces, so that they can be used to represent the add-ons (e.g. clothes) to the human body. The regular volumetric structure of our PAVM makes it easy to integrate into deep neural networks for robust surface tracking, as well as providing semantic modeling capability. The technical feasibility of the 4D reconstruction framework will be validated by the development of a Mobile 3D Facetime testbed, which will allow people in remote places to interact with each other in a natural AR fashion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

即将部署的 5G 技术使得以极低延迟的方式传输增强现实 (AR) 和混合现实 (MR) 所需的大量数据成为可能，从而实现 3D 远程沉浸式（或 3D 面对面）通信等变革性应用、使用捕获的 4D 人体内容的移动 AR 应用程序、逼真和个性化化身的人工智能助手、可以为人类服务或与人类合作的具有人类意识的机器人、将物理治疗师与他们联系起来的远程康复远离治疗设施的受伤患者等。所有这些 AR 和 MR 应用示例都需要开发实时 4D（空间和时间）捕捉和重建涉及人体、面部、身体附加物（如衣服）的动态场景以及他们周围的环境。尽管之前有关于实时 4D 重建的研究，但仍然没有开源且鲁棒的重建系统可以模拟动态场景的拓扑变化并准确且鲁棒地跟踪运动表面。该项目旨在弥合此类差距，开发一个开源且强大的 4D 重建框架，使更广泛的科学界以及行业联盟的研究人员和开发人员受益，包括 5G 医疗标准、AR 和 MR 游戏引擎、逼真的 AI 助手该项目还通过夏令营为研究生、本科生和 K-12 学生提供课程开发和教育活动。 该项目提出的研究重点是动态场景中拓扑变化的优雅建模和运动表面的鲁棒跟踪，最终目标是开发一个开源的鲁棒 4D 重建框架，用于实时捕捉动态人类场景。为了解决拓扑变化的挑战，体积融合框架及其数据结构将从根本上重新设计，通过将非流形体积网格引入截断符号距离场（TSDF）和嵌入式变形图（EDG）表示，允许体积融合细胞自我复制，边缘被破坏。这种新颖的拓扑变化感知框架将允许重建的网格几何体动态更新其连接性，以及灵活的变形图在整个 4D 捕获过程中更新其节点之间的连接性。为了解决 4D 动态人体捕捉中的鲁棒表面跟踪问题，提出了一种参数化可动画体积模型（PAVM），结合了参数化人体模型和体积 TSDF 的优点。 TSDF体积网格建立在参数化的人体表面之上，因此它们可以用来表示人体的附加物（例如衣服）。我们的 PAVM 的规则体积结构可以轻松集成到深度神经网络中，以实现强大的表面跟踪，并提供语义建模功能。 4D重建框架的技术可行性将通过移动3D Facetime测试平台的开发得到验证，该测试平台将允许偏远地区的人们以自然的AR方式相互交互。该奖项反映了NSF的法定使命，并被认为是值得的通过使用基金会的智力优势和更广泛的影响审查标准进行评估来获得支持。

项目成果

GPU-Based Supervoxel Generation With a Novel Anisotropic Metric

• DOI: 10.1109/tip.2021.3120878
• 发表时间: 2021-10
• 期刊: IEEE Transactions on Image Processing
• 影响因子: 10.6
• 作者: Xiaopan Dong;Zhonggui Chen;Yong-Jin Liu;Junfeng Yao;Xiaohu Guo

GPU-based supervoxel segmentation for 3D point clouds

• DOI: 10.1016/j.cagd.2022.102080
• 发表时间: 2022-02
• 期刊: Comput. Aided Geom. Des.
• 作者: Xiaopan Dong;Yanyang Xiao;Zhonggui Chen;Junfeng Yao;X. Guo

Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image

分层服装网络：从单个图像生成多个隐式服装层

• 发表时间: 2022
• 期刊: 2022 Asian Conference on Computer Vision (ACCV
• 作者: Aggarwal, Alakh;Wang, Jikai;Hogue, Steven;Ni, Saifeng;Budagavi, Madhukar;Guo, Xiaohu
• 通讯作者: Guo, Xiaohu

IMMAT: Mesh Reconstruction from Single View Images by Medial Axis Transform Prediction

• DOI: 10.1016/j.cad.2022.103304
• 发表时间: 2022-05
• 期刊: Comput. Aided Des.
• 作者: Jianwei Hu;Gang Chen;Baorong Yang;Ningna Wang;X. Guo;Bin Wang

Neighbor Reweighted Local Centroid for Geometric Feature Identification

• DOI: 10.1109/tvcg.2021.3124911
• 发表时间: 2021-11
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Tong Liu;Zhenhua Yang;Shaojun Hu;Zhiyi Zhang;Chunxia Xiao;Xiaohu Guo;Long Yang