RI: Small: Integrating physics, data, and art-based insights for controllable generative models

RI：小型：集成物理、数据和基于艺术的见解以实现可控生成模型

• 批准号: 2323086
• 负责人: Pavan Turaga
• 金额: $ 59.55万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2323086&HistoricalAwards=false
• 关键词: RI; Small; Integrating; physics; data

Generative models refer to a large class of machine learning techniques that can generate user-specified media – including images, video, 3D environments, and text – from inputs such as text prompts, sketches, or other user provide images. New generative models are rapidly being developed and are seen as increasingly important in many different applications such as in chatbots and automation. Current generative models are characterized by extremely large models trained on web-scale data, but on closer inspection are found to be unreliable in critically important contexts. This project focuses on generative models for visual media, where current generative models will be advanced by leveraging prior knowledge about how visual features can be described by physical and statistical laws. The sources of knowledge that will be leveraged include physics-based knowledge, insights from traditional content creation techniques, and advances in modeling latent-spaces using novel geometric methods. The anticipated benefits include more robust models, smaller scale models, and more interpretable and modular models. This research systematically investigating the basics of generative-adversarial networks. The first task considers the role of the input probability distribution from which samples are drawn, generalizing to non-parametric distributions tuned to reduce distribution mismatch under sample mixing. The second task involves architectural novelty in terms of detail layering, where synthesis is broken into a series of simpler architectures. The third task focuses on developing reduced parameter discriminator models, using orthogonality-type constraints as a proxy for physical variables like lighting, texture, and deformation. The fourth task focuses on developing shape-aware architectures, using learnable polynomial basis functions to represent shape more directly. Applications for these methods include augmenting training-sets to create trustworthy machine learning models in contexts such as manufacturing and health, where it is difficult to gather large training sets. Curricular innovations include creating access to these approaches for non-STEM students, in a class titled Machine Learning for Media Arts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

通用模型是指从输入（例如文本提示，草图或其他用户提供图像）的输入中，该技术可以生成用户指定的媒体（包括图像，视频，3D环境和文本），其中包括图像，视频，3D环境和文本。新的通用模型正在迅速开发，并且在许多不同的应用程序（例如聊天机器人和自动化）中被认为越来越重要。当前的通用模型的特征是在网络尺度数据上训练的非常大的模型，但在仔细检查时，发现在至关重要的环境中是不可靠的。该项目着重于视觉媒体的通用模型，在该模型中，当前通用模型将通过利用有关如何通过物理和统计定律描述视觉特征的先验知识来提出。将要利用的知识来源包括基于物理的知识，传统内容创建技术的见解以及使用新颖的几何方法对潜在空间进行建模的进步。预期的好处包括更健壮的模型，较小的规模模型以及更容易解释和模块化的模型。这项研究系统地研究了通用对流网络的基础知识。第一个任务考虑了从中绘制样品的输入概率分布的作用，从而推广到调谐的非参数分布，以减少样品混合下的分布不匹配。第二个任务涉及在细节分层方面的建筑新颖性，其中综合分为一系列简单的体系结构。第三个任务着重于开发减少参数歧视模型，使用正交性型约束作为照明，纹理和变形等物理变量的代理。第四任任务着重于开发形状感知体系结构，使用可学习的多项式基础函数来更直接地表示形状。这些方法的应用包括增加培训集，以在制造和健康等环境中创建值得信赖的机器学习模型，在那里很难收集大型培训集。课程创新包括在一个名为“媒体艺术的机器学习”的课堂上为非茎学生创建这些方法的访问权。该奖项反映了NSF的法定使命，并使用基金会的知识分子优点和更广泛的影响评估标准，被视为通过评估来获得珍贵的支持。