Immersive Multisensory Media: from Quality of Experience to Quality of Life

沉浸式多感官媒体：从体验质量到生活质量

• 批准号: RGPIN-2021-03246
• 负责人: Falk, Tiago
• 金额: $ 2.84万
• 依托单位: Institut national de la recherche scientifique
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=742450
• 关键词: Immersive; Multisensory; Media; Quality; Experience

Immersive communication aims to enable natural experiences and interactions with remote people and environments in ways that blur the line between real and virtual environments. Today, our need for more effective ways to communicate with remote people has never been greater. Existing immersive technologies and applications, however, rely only on two senses: vision and hearing. To enable true immersive communications, we need to involve other senses. Imagine yourself watching a movie where you feel the heat from an explosion or a TV ad where you can smell the food. With advances in 5G, the Tactile Internet, and neurotechnology, the next decade will see the Internet of Senses revolution, where mind, touch, smells, taste, sight, and sound will come together. Ultimately, the success or failure of any new immersive technology will rely on the human experience it provides, and not on the technology it uses. Human factors, such as sense of presence, immersion, cybersickness, and cognitive states all contribute to the user's perceived immersive media experience (IMEx). In turn, objective (instrumental) measurement of these factors could allow for immersive content and sensory stimulation to be updated in real-time, thus maximizing the per-user experience. Measurement of such human factors, however, is extremely challenging and this is where the proposed Research Program comes in. In particular, several short-term objectives have been defined, grouped under three main themes. First, we propose to develop instrumental measures of several factors, including presence, immersion, cybersickness, and emotion, and then gauge their effects on overall perceived audio-visual IMEx. We propose a combination of artifact-robust psychophysiological and behavioural metrics, biosignal enhancement/compression techniques, and user-aware machine learning algorithms. Applications for single and multiple users working collaboratively will be developed, thus building on emerging hyperscanning innovations in cross-subject synchrony and task performance. Second, we propose to expand the developed tools to encompass multisensory immersive media, where haptics and smells will be combined. Lastly, the developed algorithms will all come together and two multisensory applications to improve the quality of life of Canadians will be developed: neurofeedback for anxiety treatment and Alzheimer's disease progression monitoring. The proposed research program will emphasize excellence in training of highly qualified personnel (HQP), who will be part of an inclusive, diverse, and equitable training environment. Given the cross-disciplinary nature of the program, HQP will be exposed to a mix of multisensory signal processing, serious gaming, human factors, and machine learning, thus equipping them with skills and expertise that are in high demand in today's competitive job market. The research program will provide significant economical, technological and societal benefits to Canada.

沉浸式通信旨在以模糊真实和虚拟环境之间界限的方式实现与远程人员和环境的自然体验和交互。如今，我们对与远程人员进行更有效沟通的需求从未如此强烈。然而，现有的沉浸式技术和应用仅依赖于两种感官：视觉和听觉。为了实现真正的沉浸式交流，我们需要调动其他感官。想象一下，您在观看电影时会感受到爆炸带来的热量，或者在观看电视广告时会闻到食物的味道。随着 5G、触觉互联网和神经技术的进步，未来十年将迎来感官互联网革命，思想、触觉、嗅觉、味觉、视觉和声音将融合在一起。最终，任何新的沉浸式技术的成功或失败都将取决于它提供的人类体验，而不是它使用的技术。存在感、沉浸感、晕机症和认知状态等人为因素都会影响用户感知的沉浸式媒体体验 (IMEx)。反过来，对这些因素的客观（仪器）测量可以允许实时更新沉浸式内容和感官刺激，从而最大化每个用户的体验。然而，对此类人为因素的衡量极具挑战性，这就是拟议的研究计划的用武之地。特别是，已经定义了几个短期目标，分为三个主题。首先，我们建议开发几个因素的工具测量，包括存在感、沉浸感、晕机症和情绪，然后评估它们对整体感知视听 IMEx 的影响。我们提出了将人工制品稳健的心理生理学和行为指标、生物信号增强/压缩技术和用户感知的机器学习算法相结合。将开发用于单个和多个用户协作工作的应用程序，从而建立在跨主题同步和任务性能方面新兴的超扫描创新之上。其次，我们建议扩展已开发的工具，以涵盖多感官沉浸式媒体，其中触觉和气味将结合起来。最后，所开发的算法将全部整合在一起，并将开发两个多感官应用程序来提高加拿大人的生活质量：用于焦虑治疗的神经反馈和阿尔茨海默病进展监测。 拟议的研究计划将强调高素质人才（HQP）的卓越培训，他们将成为包容、多元化和公平培训环境的一部分。鉴于该计划的跨学科性质，HQP 将接触多感官信号处理、严肃游戏、人为因素和机器学习，从而使他们具备当今竞争激烈的就业市场急需的技能和专业知识。该研究计划将为加拿大带来显着的经济、技术和社会效益。