CHS: Small: Appearance Modeling by Synthesis

CHS：小型：综合外观建模

• 批准号: 1909028
• 负责人: Pieter Peers
• 金额: $ 49.99万
• 依托单位: College of William and Mary
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909028&HistoricalAwards=false
• 关键词: CHS; Small; Appearance; Modeling; Synthesis

Appearance modeling aims to create digital representations of materials, ranging from imaginary materials drawn by artists to reproductions of physical materials. To date, the most successful methods for creating realistic digital materials have been data-driven, where data captured from a physical material sample is used to reconstruct a digital representation. Most prior work assumes that the measurements fully constrain the reconstruction process. This project will explore a new paradigm that views appearance modeling as a constrained synthesis process instead of a reconstruction / interpolation process. This change in paradigm requires a rethinking of fundamental appearance modeling assumptions. First, constrained synthesis produces a distribution of possible solutions rather than a single most likely solution as in classic appearance modeling methods. This creates opportunities for users to participate in the appearance modeling process (for example, by selecting the "best"' solution according to subjective or artistic criteria). Furthermore, instead of building up a solution with additional appearance measurements, constrained synthesis reduces the solution distribution with each additional measurement; that is to say, it promotes a subtractive approach to appearance modeling. A constrained synthesis approach to appearance modeling also offers an elegant and scalable solution to reproducing a material's appearance from insufficient or incomplete data. This research builds on methods from computer vision and machine learning, and has the potential to advance the state-of-the-art in both fields. More broadly, the reconstruction and synthesis methods developed will be applicable to fields that model high dimensional data and for which it is difficult to obtain samples. The results from the proposed research activities will be incorporated in new and existing courses, recruitment activities at the graduate and undergraduate level, and outreach activities promoting STEM to minorities.This project will advance constrained synthesis as a new paradigm for appearance modeling through three research thrusts that explore generative adversarial networks (GANs) for: (1) unconstrained material generators, (2) constrained material generators, and (3) empowering users to author new material generators. In each of these research thrusts, the concept of a (potentially non-linear) projection that provides a mapping from the high dimensional target distribution to one or more lower-dimensional source distributions plays a central role. Associating a discriminator with each projection yields novel GAN architectures that decouple the spaces of the training data, conditions, and the target distribution. Building on the concept of projections and the resulting decoupling of distribution spaces, each of the three research thrusts endeavors to contribute to a different subfield in appearance modeling: (1) synthesizing novel instances of spatially varying materials, (2) reconstructing a material from a variable number of observations or from incomplete measurements, and (3) designing novel material generators by restricting the output of a general material generator based on user preferences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

外观建模旨在创建材料的数字表示，从艺术家绘制的虚构材料到物理材料的复制品。 迄今为止，创建现实数字材料的最成功方法是数据驱动的，其中使用物理材料样本捕获的数据用于重建数字表示。大多数先前的工作都假定测量结果完全限制了重建过程。 该项目将探索一个新的范式，该范式将外观建模视为受约束的合成过程，而不是重建 /插值过程。 这种范式的变化需要重新思考基本的外观建模假设。 首先，受约束的合成产生了可能的解决方案的分布，而不是像经典外观建模方法一样最有可能的解决方案。这为用户创造了参与外观建模过程的机会（例如，根据主观或艺术标准选择“最佳”解决方案）。 此外，限制的合成不是用其他外观测量来建立溶液，而是通过每次额外的测量来降低溶液分布。也就是说，它促进了外观建模的减法方法。 外观建模的约束合成方法还提供了一种优雅的可扩展解决方案，可从不足或不完整的数据中重现材料的外观。这项研究基于计算机视觉和机器学习的方法，并有可能推进这两个领域的最新技术。更广泛地，开发的重建和合成方法将适用于模拟高维数据且难以获得样品的字段。 The results from the proposed research activities will be incorporated in new and existing courses, recruitment activities at the graduate and undergraduate level, and outreach activities promoting STEM to minorities.This project will advance constrained synthesis as a new paradigm for appearance modeling through three research thrusts that explore generative adversarial networks (GANs) for: (1) unconstrained material generators, (2) constrained material generators, and (3) empowering users to作者新材料生成器。 在这些研究中的每一个中，（潜在的非线性）投影的概念提供了从高维目标分布到一个或多个较低维源分布的映射，起着核心作用。 将一个歧视者与每个投影相关联会产生新型的gan体系结构，从而使训练数据，条件和目标分布的空间分离。 基于预测概念和由此产生的分配空间的脱成耦合，这三项研究都在努力促进外观模型中的不同子场，（1）合成空间变化材料的新颖实例，（2）从可变化的材料数量或从不良的材料中重建材料的材料，从而构建了一般的材料，并通过不合时宜的材料（3）设计材料（3）材料生成器（3）材料生成器（3）材料的生成器（3）材料的生成器（3）偏好。该奖项反映了NSF的法定使命，并通过使用基金会的知识分子优点和更广泛的影响审查标准来评估值得支持。

项目成果

An Adaptive BRDF Fitting Metric

自适应 BRDF 拟合指标

• DOI: 10.1111/cgf.14054
• 发表时间: 2020
• 期刊: Computer Graphics Forum
• 影响因子: 2.5
• 作者: Bieron, J.;Peers, P.
• 通讯作者: Peers, P.

Glass Segmentation using Intensity and Spectral Polarization Cues

• DOI: 10.1109/cvpr52688.2022.01229
• 发表时间: 2022-01-01
• 期刊: 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)
• 作者: Mei, Haiyang;Dong, Bo;Yang, Xin
• 通讯作者: Yang, Xin

Deep Reflectance Scanning: Recovering Spatially‐varying Material Appearance from a Flash‐lit Video Sequence

• DOI: 10.1111/cgf.14387
• 发表时间: 2021-08
• 期刊: Computer Graphics Forum
• 影响因子: 2.5
• 作者: Wenjie Ye;Yue Dong;P. Peers;B. Guo

Depth-Aware Mirror Segmentation

• DOI: 10.1109/cvpr46437.2021.00306
• 发表时间: 2021-06
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Haiyang Mei;Bo Dong;Wen Dong;P. Peers;Xin Yang;Qiang Zhang;Xiaopeng Wei

Interactive Curation of Datasets for Training and Refining Generative Models

• DOI: 10.1111/cgf.13844
• 发表时间: 2019-10
• 期刊: Computer Graphics Forum
• 影响因子: 2.5
• 作者: Wenjie Ye;Yue Dong;P. Peers