Multimodal sparsity models for segmentation of visual objects

用于视觉对象分割的多模态稀疏模型

• 批准号: 263894508
• 负责人: Professor Dr. Daniel Cremers
• 金额: --
• 依托单位: Lehrstuhl für Datenverarbeitung (LDV)
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2015
• 资助国家: 德国
• 起止时间: 2014-12-31 至 2020-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/263894508?language=en
• 关键词: Multimodal; sparsity; models; segmentation; visual

Automatic segmentation of objects in a 3D scene is a fundamental problem in Computer Vision. It is a core aspect for many high-level algorithms including object recognition, semantic scene analysis and 3D object reconstruction. To date unsupervised segmentation faces many challenges such as complex texture and lighting variations in real-world scenes. The proposed project is based on the reasonable assumption that an object is determined by characteristic properties of signals captured through different modalities. We intend to demonstrate that a rigorous mathematical derivation of multimodal segmentation approaches will lead to drastic improvements in unsupervised segmentation ultimately allowing for a robust and precise segmentation of physical objects in the scene. More specifically we will focus on the following challenges: - Inspired by biological systems which perceive their environment through many different signal modalities at once, we intend to devise algorithms to fuse sensory information from different modalities in order to drastically enhance the performance of unsupervised segmentation methods. - Unlike existing fusion schemes that combine information on higher processing levels, we will focus on fusion across several modalities on the signal level.- We will generalize existing techniques for sparse representation and inference from the unimodal setting to the multimodal setting and demonstrate that an accurate modeling of sparsity and interdependency of the multiple channels allows for much better discrimination of objects of interest. Our methods will be general enough to work on different types of visual data and their cross-modal dependencies, without the need of hand-crafted signal-specific features. We investigate sparse signal representations in general and focus on the so-called co-sparse analysis model in particular. - We will derive variational segmentation algorithms which exploit multimodal sparsity for unsupervised object segmentation, thereby combining multimodal sparsity with powerful convex regularization methods.- We will develop a demonstrator which generates semantic 3D segmentations from multimodal data taken from multiple views of a scene. We hope to demonstrate that the proposed method is robust to noise and challenging lighting conditions of real-world environments.

3D 场景中对象的自动分割是计算机视觉中的一个基本问题。它是许多高级算法的核心部分，包括对象识别、语义场景分析和 3D 对象重建。迄今为止，无监督分割面临着许多挑战，例如现实场景中复杂的纹理和光照变化。所提出的项目基于合理的假设，即物体是由通过不同方式捕获的信号的特征属性确定的。我们打算证明，多模态分割方法的严格数学推导将导致无监督分割的巨大改进，最终允许对场景中的物理对象进行稳健且精确的分割。更具体地说，我们将重点关注以下挑战： - 受到同时通过多种不同信号模态感知环境的生物系统的启发，我们打算设计算法来融合来自不同模态的感官信息，以大幅提高无监督分割方法的性能。 - 与结合更高处理级别信息的现有融合方案不同，我们将重点关注信号级别上多种模态的融合。 - 我们将推广稀疏表示和推理的现有技术，从单模态设置到多模态设置，并证明准确的对多个通道的稀疏性和相互依赖性进行建模可以更好地区分感兴趣的对象。我们的方法将足够通用，可以处理不同类型的视觉数据及其跨模式依赖性，而不需要手工制作的信号特定特征。我们一般研究稀疏信号表示，并特别关注所谓的协同稀疏分析模型。 - 我们将推导出利用多模态稀疏性进行无监督对象分割的变分分割算法，从而将多模态稀疏性与强大的凸正则化方法相结合。- 我们将开发一个演示器，该演示器可以根据从场景的多个视图获取的多模态数据生成语义 3D 分割。我们希望证明所提出的方法对于现实环境中的噪声和具有挑战性的照明条件具有鲁棒性。

项目成果

Co-Sparse Textural Similarity for Interactive Segmentation

用于交互式分割的共稀疏纹理相似度

• DOI: 10.1007/978-3-319-10599-4_19
• 发表时间: 2014
• 作者: Claudia Nieuwenhuis; Simon Hawe; Martin Kleinsteuber; Daniel Cremers
• 通讯作者: Daniel Cremers

Noise-resistant Unsupervised Object Segmentation in Multi-view Indoor Point Clouds

多视图室内点云中的抗噪声无监督对象分割

• DOI: 10.5220/0006100801490156
• 作者: Dmytro Bobkov; Sili Chen; Martin Kiechle; Simon Hilsenbeck; Eckehard Steinbach
• 通讯作者: Eckehard Steinbach

CPA-SLAM: Consistent plane-model alignment for direct RGB-D SLAM

CPA-SLAM：直接 RGB-D SLAM 的一致平面模型对齐

• DOI: 10.1109/icra.2016.7487260
• 发表时间: 2016-05-16
• 期刊: 2016 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: Lingni Ma;C. Kerl;J. Stückler;D. Cremers
• 通讯作者: D. Cremers

Multi-view deep learning for consistent semantic mapping with RGB-D cameras

使用 RGB-D 相机实现一致语义映射的多视图深度学习

• DOI: 10.1109/iros.2017.8202213
• 发表时间: 2017-03-26
• 期刊: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
• 作者: Lingni Ma;J. Stückler;C. Kerl;D. Cremers
• 通讯作者: D. Cremers

Room segmentation in 3D point clouds using anisotropic potential fields

使用各向异性势场在 3D 点云中进行房间分割

• DOI: 10.1109/icme.2017.8019484
• 发表时间: 2017
• 期刊: 2017 IEEE International Conference on Multimedia and Expo (ICME)
• 作者: Dmytro Bobkov; Martin Kiechle; Simon Hilsenbeck; Eckehard Steinbach
• 通讯作者: Eckehard Steinbach