CHS: Small: Appearance Modeling by Synthesis

CHS：小型：综合外观建模

• 批准号: 1909028
• 负责人: Pieter Peers
• 金额: $ 49.99万
• 依托单位: College of William and Mary
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909028&HistoricalAwards=false
• 关键词: CHS; Small; Appearance; Modeling; Synthesis

Appearance modeling aims to create digital representations of materials, ranging from imaginary materials drawn by artists to reproductions of physical materials. To date, the most successful methods for creating realistic digital materials have been data-driven, where data captured from a physical material sample is used to reconstruct a digital representation. Most prior work assumes that the measurements fully constrain the reconstruction process. This project will explore a new paradigm that views appearance modeling as a constrained synthesis process instead of a reconstruction / interpolation process. This change in paradigm requires a rethinking of fundamental appearance modeling assumptions. First, constrained synthesis produces a distribution of possible solutions rather than a single most likely solution as in classic appearance modeling methods. This creates opportunities for users to participate in the appearance modeling process (for example, by selecting the "best"' solution according to subjective or artistic criteria). Furthermore, instead of building up a solution with additional appearance measurements, constrained synthesis reduces the solution distribution with each additional measurement; that is to say, it promotes a subtractive approach to appearance modeling. A constrained synthesis approach to appearance modeling also offers an elegant and scalable solution to reproducing a material's appearance from insufficient or incomplete data. This research builds on methods from computer vision and machine learning, and has the potential to advance the state-of-the-art in both fields. More broadly, the reconstruction and synthesis methods developed will be applicable to fields that model high dimensional data and for which it is difficult to obtain samples. The results from the proposed research activities will be incorporated in new and existing courses, recruitment activities at the graduate and undergraduate level, and outreach activities promoting STEM to minorities.This project will advance constrained synthesis as a new paradigm for appearance modeling through three research thrusts that explore generative adversarial networks (GANs) for: (1) unconstrained material generators, (2) constrained material generators, and (3) empowering users to author new material generators. In each of these research thrusts, the concept of a (potentially non-linear) projection that provides a mapping from the high dimensional target distribution to one or more lower-dimensional source distributions plays a central role. Associating a discriminator with each projection yields novel GAN architectures that decouple the spaces of the training data, conditions, and the target distribution. Building on the concept of projections and the resulting decoupling of distribution spaces, each of the three research thrusts endeavors to contribute to a different subfield in appearance modeling: (1) synthesizing novel instances of spatially varying materials, (2) reconstructing a material from a variable number of observations or from incomplete measurements, and (3) designing novel material generators by restricting the output of a general material generator based on user preferences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

外观建模旨在创建材料的数字表示，范围从艺术家绘制的想象材料到物理材料的复制品。 迄今为止，创建真实数字材料的最成功的方法是数据驱动的，其中从物理材料样本捕获的数据用于重建数字表示。大多数先前的工作都假设测量完全限制了重建过程。 该项目将探索一种新的范例，将外观建模视为受约束的合成过程，而不是重建/插值过程。 这种范式的变化需要重新思考基本的外观建模假设。 首先，约束综合产生可能解决方案的分布，而不是像经典外观建模方法那样产生单个最可能的解决方案。这为用户创造了参与外观建模过程的机会（例如，通过根据主观或艺术标准选择“最佳”解决方案）。 此外，约束合成不是通过额外的外观测量来构建解决方案，而是通过每个额外的测量来减少解决方案的分布；也就是说，它提倡一种减法的外观建模方法。 外观建模的约束合成方法还提供了一种优雅且可扩展的解决方案，可以根据不充分或不完整的数据再现材料的外观。这项研究建立在计算机视觉和机器学习方法的基础上，有潜力推动这两个领域的最新技术发展。更广泛地说，所开发的重建和合成方法将适用于对高维数据进行建模且难以获取样本的领域。 拟议研究活动的结果将纳入新的和现有的课程、研究生和本科生的招聘活动以及向少数族裔推广 STEM 的外展活动中。该项目将通过三个研究重点推进约束合成作为外观建模的新范式探索生成对抗网络 (GAN)：(1) 无约束材料生成器，(2) 受约束材料生成器，以及 (3) 使用户能够创作新材料生成器。 在这些研究的每一个主旨中，提供从高维目标分布到一个或多个低维源分布的映射的（可能是非线性的）投影概念发挥着核心作用。 将鉴别器与每个投影相关联会产生新颖的 ​​GAN 架构，该架构可以解耦训练数据、条件和目标分布的空间。 基于投影的概念和由此产生的分布空间解耦，这三个研究主旨中的每一个都致力于为外观建模中的不同子领域做出贡献：（1）合成空间变化材料的新颖实例，（2）从可变数量的观察或不完整的测量，以及（3）通过根据用户偏好限制通用材料生成器的输出来设计新颖的材料生成器。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准。

项目成果

An Adaptive BRDF Fitting Metric

自适应 BRDF 拟合指标

• DOI: 10.1111/cgf.14054
• 发表时间: 2020
• 期刊: Computer Graphics Forum
• 影响因子: 2.5
• 作者: Bieron, J.;Peers, P.
• 通讯作者: Peers, P.

Glass Segmentation using Intensity and Spectral Polarization Cues

• DOI: 10.1109/cvpr52688.2022.01229
• 发表时间: 2022-01-01
• 期刊: 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)
• 作者: Mei, Haiyang;Dong, Bo;Yang, Xin
• 通讯作者: Yang, Xin

Deep Reflectance Scanning: Recovering Spatially‐varying Material Appearance from a Flash‐lit Video Sequence

• DOI: 10.1111/cgf.14387
• 发表时间: 2021-08
• 期刊: Computer Graphics Forum
• 影响因子: 2.5
• 作者: Wenjie Ye;Yue Dong;P. Peers;B. Guo

Depth-Aware Mirror Segmentation

• DOI: 10.1109/cvpr46437.2021.00306
• 发表时间: 2021-06
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Haiyang Mei;Bo Dong;Wen Dong;P. Peers;Xin Yang;Qiang Zhang;Xiaopeng Wei

Interactive Curation of Datasets for Training and Refining Generative Models

• DOI: 10.1111/cgf.13844
• 发表时间: 2019-10
• 期刊: Computer Graphics Forum
• 影响因子: 2.5
• 作者: Wenjie Ye;Yue Dong;P. Peers