OAC Core: Small: Open-Source Robust 4D Reconstruction Framework for Real-Time Dynamic Human Capture

OAC Core：小型：用于实时动态人体捕捉的开源稳健 4D 重建框架

• 批准号: 2007661
• 负责人: Xiaohu Guo
• 金额: $ 50万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007661&HistoricalAwards=false
• 关键词: OAC; Core; Small; Open; Source

The upcoming deployment of 5G technology makes it feasible to communicate with extremely low latency the vast amounts of data needed for Augmented Reality (AR) and Mixed Reality (MR), which enables transformative applications such as 3D tele-immersive (or 3D facetime) communication, mobile AR apps using captured 4D human contents, AI assistants of lifelike and personalized avatars, human-aware robots that can serve or work with humans, tele-rehabilitation to connect physical therapists with wounded patients far from treatment facilities, etc. All these examples of AR and MR applications require the development of real-time 4D (space and time) capture and reconstruction of dynamic scenes involving human bodies, faces, body add-ons (like clothes), and their surrounding environments. Despite prior research on real-time 4D reconstruction, there is still no open-source and robust reconstruction system that can model topological changes of the dynamic scenes and track the moving surfaces with accuracy and robustness. This project is designed to bridge such gaps, to develop an open-source and robust 4D reconstruction framework that can benefit researchers and developers in broader scientific communities as well as industry alliances, including 5G medical standards, AR and MR game engines, lifelike AI assistants, human-aware robotics, tele-rehabilitation, etc. This project also provides curriculum development and educational activities for graduate, undergraduate, and K-12 students through summer camps. The research proposed for this project centers around the elegant modeling of topological changes in dynamic scenes and the robust tracking of moving surfaces, towards the ultimate goal of developing an open-source robust 4D reconstruction framework for real-time capture of dynamic human scenes. To solve the challenges of topological changes, the volumetric fusion framework and its data structures will be fundamentally redesigned, by introducing Non-manifold Volumetric Grids into both Truncated Signed Distance Field (TSDF) and Embedded Deformation Graph (EDG) representations, allowing both the volumetric cells to replicate themselves and the edges to be broken. Such a novel topology-change-aware framework will allow the reconstructed mesh geometry to update its connectivity on-the-fly, along with a flexible deformation graph updating its connectivity between nodes throughout the 4D capture process. To solve the robust surface tracking problem in 4D dynamic human capture, a Parameterized Animatable Volumetric Model (PAVM) is proposed to combine the benefits of both the parametric human body model and the volumetric TSDF. The TSDF volumetric grids are built on top of the parameterized human body surfaces, so that they can be used to represent the add-ons (e.g. clothes) to the human body. The regular volumetric structure of our PAVM makes it easy to integrate into deep neural networks for robust surface tracking, as well as providing semantic modeling capability. The technical feasibility of the 4D reconstruction framework will be validated by the development of a Mobile 3D Facetime testbed, which will allow people in remote places to interact with each other in a natural AR fashion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The upcoming deployment of 5G technology makes it feasible to communicate with extremely low latency the vast amounts of data needed for Augmented Reality (AR) and Mixed Reality (MR), which enables transformative applications such as 3D tele-immersive (or 3D facetime) communication, mobile AR apps using captured 4D human contents, AI assistants of lifelike and personalized avatars, human-aware robots that can serve or work with humans,将物理治疗师与受伤的患者连接到远离治疗设施的远程治疗。所有这些AR和MR应用的例子都需要开发实时4D（时空）捕获和重建涉及人体，面部，身体附加组件（如衣服）及其周围环境的动态场景的动态场景。尽管事先研究了实时4D重建，但仍然没有开源和健壮的重建系统可以对动态场景的拓扑变化进行建模，并以准确性和鲁棒性跟踪移动表面。 This project is designed to bridge such gaps, to develop an open-source and robust 4D reconstruction framework that can benefit researchers and developers in broader scientific communities as well as industry alliances, including 5G medical standards, AR and MR game engines, lifelike AI assistants, human-aware robotics, tele-rehabilitation, etc. This project also provides curriculum development and educational activities for graduate, undergraduate, and K-12 students through summer营地。 该项目提出的研究集中在动态场景中拓扑变化的优雅建模以及对移动表面的稳健跟踪的最终目标，这是开发开源稳健的4D重建框架，以实时捕捉动态人类场景。为了解决拓扑变化的挑战，通过将非Manifold的量大网格引入截断的签名距离字段（TSDF）（TSDF）和嵌入式变形图（EDG）表示形式，从根本上重新设计了体积融合框架及其数据结构，从而可以重新设计。这样一种新颖的拓扑变更感知框架将允许重建的网格几何形状在正面更新其连接性，以及灵活的变形图，在4D捕获过程中更新节点之间的连接性。为了解决4D动态人类捕获中强大的表面跟踪问题，提出了一个参数化的动画体积模型（PAVM），以结合参数人体模型和体积TSDF的好处。 TSDF体积网格建立在参数化的人体表面之上，因此可以用来代表人体的附加组件（例如衣服）。我们PAVM的常规体积结构使整合到深层神经网络中，以进行稳健的表面跟踪，并提供语义建模能力。 4D重建框架的技术可行性将通过开发3D FaceTime测试床的开发来验证，这将使偏远地区的人们以自然的AR方式相互互动。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的智力和更广泛影响的评估来通过评估来获得支持的人。

项目成果

GPU-Based Supervoxel Generation With a Novel Anisotropic Metric

• DOI: 10.1109/tip.2021.3120878
• 发表时间: 2021-10
• 期刊: IEEE Transactions on Image Processing
• 影响因子: 10.6
• 作者: Xiaopan Dong;Zhonggui Chen;Yong-Jin Liu;Junfeng Yao;Xiaohu Guo

GPU-based supervoxel segmentation for 3D point clouds

• DOI: 10.1016/j.cagd.2022.102080
• 发表时间: 2022-02
• 期刊: Comput. Aided Geom. Des.
• 作者: Xiaopan Dong;Yanyang Xiao;Zhonggui Chen;Junfeng Yao;X. Guo

Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image

分层服装网络：从单个图像生成多个隐式服装层

• 发表时间: 2022
• 期刊: 2022 Asian Conference on Computer Vision (ACCV
• 作者: Aggarwal, Alakh;Wang, Jikai;Hogue, Steven;Ni, Saifeng;Budagavi, Madhukar;Guo, Xiaohu
• 通讯作者: Guo, Xiaohu

IMMAT: Mesh Reconstruction from Single View Images by Medial Axis Transform Prediction

• DOI: 10.1016/j.cad.2022.103304
• 发表时间: 2022-05
• 期刊: Comput. Aided Des.
• 作者: Jianwei Hu;Gang Chen;Baorong Yang;Ningna Wang;X. Guo;Bin Wang

Neighbor Reweighted Local Centroid for Geometric Feature Identification

• DOI: 10.1109/tvcg.2021.3124911
• 发表时间: 2021-11
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Tong Liu;Zhenhua Yang;Shaojun Hu;Zhiyi Zhang;Chunxia Xiao;Xiaohu Guo;Long Yang