Scene Processing with Machine Learnable and Semantically Parametrized Scene Representations

使用机器学习和语义参数化场景表示进行场景处理

• 批准号: MR/T043229/1
• 负责人: Ahmet Oztireli
• 金额: $ 154.77万
• 依托单位: University of Cambridge
• 依托单位国家: 英国
• 项目类别: Fellowship
• 财政年份: 2021
• 资助国家: 英国
• 起止时间: 2021 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=MR%2FT043229%2F1
• 关键词: Scene; Processing; Machine; Learnable; Semantically

Digital representations of visual reality and imagination is an integral part of almost all scientific disciplines, industry, and society. Imaging techniques and computer graphics have successfully solved the problems of creating accurate projections, e.g. image and video, of the real world over the last decades. However, such projections are fundamentally limited representations of the underlying scenes and only allow for passive consumption such as viewing on a screen. Many applications from image editing to augmented/virtual reality instead require active 3D exploration, creation, and editing of scenes, for which we need full virtual scene models. Creating virtual scenes that are high fidelity models of the real world and beyond via digital modelling or capture has traditionally been a privilege only available to corporations with educated artists and engineers, working with complex software and hardware tools for countless hours. These professionals produce and work with carefully designed parametrizations of geometry, appearance, and motion, which allow them to author and edit virtual scenes with all the intricate details of reality and their imagination. On the other end, many computer vision techniques try to have a shortcut by capturing scenes from the real world via simpler sensors in uncontrolled environments, or altering images of scenes with algorithms that only implicitly encode scene semantics, e.g. in latent spaces of artificial neural networks. These lead to scene representations of lower quality and in a form that is not easily editable for semantically meaningful modelling. The objective of this research project is to tackle these shortcomings and develop a scene representation that is 1) a high fidelity detailed model of visual reality in terms of geometry, appearance, and motion, 2) machine learnable via capture in partially controlled practical environments, 3) semantically parametrized to allow for easy and intuitive edits, 4) fast to visualize for real-time exploration. Based on this representation, we will develop scene processing techniques that will allow individuals to create, alter, explore, and share high fidelity virtual objects and scenes, unlocking a completely new set of applications in augmented/virtual reality, gaming, product design, manufacturing, education, robotics, and medical domains.

视觉现实和想象力的数字表示是几乎所有科学学科，工业和社会的组成部分。成像技术和计算机图形成功解决了创建准确预测的问题，例如过去几十年来，现实世界的图像和视频。但是，此类预测从根本上是基础场景的有限表示，并且仅允许被动消费，例如在屏幕上查看。从图像编辑到增强/虚拟现实的许多应用都需要主动的3D探索，创建和对场景的编辑，我们需要完整的虚拟场景模型。传统上，创建虚拟场景是现实世界中的高保真模型以及通过数字建模或捕获的超越模型，这是一种特权，只有具有教育的艺术家和工程师的公司才能使用，使用了无数小时的复杂软件和硬件工具。这些专业人员通过精心设计的几何形状，外观和运动的参数来生产和工作，这使他们能够用现实的所有复杂细节及其想象力创作和编辑虚拟场景。另一方面，许多计算机视觉技术试图通过在不受控制的环境中通过较简单的传感器捕获现实世界的场景，或使用算法更改场景的图像，而算法仅隐含地编码场景语义，例如。在人工神经网络的潜在空间中。这些导致质量较低和形式的场景表示形式，对于语义上有意义的建模不容易编辑。该研究项目的目的是解决这些缺点，并开发一个场景表示，即1）在几何，外观和运动方面，高保真详细的视觉现实模型，2）通过在部分控制的实践环境中通过捕获来学习的机器，3）语义参数化，以允许轻松和直觉的编辑，4）快速进行实时探索。基于此表示形式，我们将开发场景处理技术，使个人能够创建，更改，探索和共享高保真虚拟对象和场景，从而在增强/虚拟现实，游戏，产品设计，制造，教育，机器人和医学领域中解锁全新的应用程序。

项目成果

Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

• DOI: 10.1109/cvpr46437.2021.00044
• 发表时间: 2020-12
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Yifan Wang;Shihao Wu;A. C. Öztireli;O. Sorkine-Hornung

Kubric: A scalable dataset generator

• DOI: 10.1109/cvpr52688.2022.00373
• 发表时间: 2022-01-01
• 期刊: 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022)
• 作者: Greff, Klaus;Belletti, Francois;Tagliasacchi, Andrea
• 通讯作者: Tagliasacchi, Andrea

Inferring implicit 3D representations from human figures on pictorial maps

从图画地图上的人物推断隐式 3D 表示

• DOI: 10.1080/15230406.2023.2224063
• 发表时间: 2023
• 期刊: Cartography and Geographic Information Science
• 影响因子: 2.5
• 作者: Schnürer R

Differentiable visual computing for inverse problems and machine learning

• DOI: 10.1038/s42256-023-00743-0
• 发表时间: 2023-11
• 期刊: Nature Machine Intelligence
• 影响因子: 23.8
• 作者: Andrew Spielberg;Fangcheng Zhong;Konstantinos Rematas;Krishna Murthy Jatavallabhula;Cengiz Oztireli;Tzu-Mao Li;Derek Nowrouzezahrai

Statistical shape representations for temporal registration of plant components in 3D

用于 3D 植物组件时间配准的统计形状表示

• DOI: 10.1109/icra48891.2023.10160709
• 发表时间: 2023
• 作者: Heiwolt K