NetSE:Large:Collaborative Research: Exploiting Multi-modality for Tele-Immersion

NetSE：大型：协作研究：利用多模态实现远程沉浸

• 批准号: 1012975
• 负责人: Balakrishnan Prabhakaran
• 金额: $ 203.38万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-10-01 至 2018-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1012975&HistoricalAwards=false
• 关键词: NetSE; Large; Collaborative; Research; Exploiting

Providing an environment that offers both immersion and interaction is a tough research challenge. Ensuring a reasonable Quality of Experience (QoE) in using these environments installed in geographically distributed cities is even a tougher challenge. This project considers a collaborative, immersive, and interactive environment that not only supports 3D rendering of the participants? video but also other modalities such as Body Sensor Network (BSN) data that can offer highly precise data about a person?s physical movements (as well as physiological data). While creating this environment, one needs to consider the various bottlenecks that choke the data streams carrying the immersive and interactive information: reconstruction delay, ultra-high throughput needed, packet loss, and rendering delays. The main aim of this project is to design and develop collaborative, multi-modal immersive environments with higher frame rates and frame quality by carrying out research tasks that can take advantage of information from other modalities and handle these bottlenecks.In a typical tele-immersive environment, participants can see themselves in the locally rendered 3D view and see participants in the remote environments as well. Since the local rendering delays are much smaller, participants can see themselves earlier and in a more smooth fashion compared to the rendering of remote participants that suffers from communication delays and packet losses. This aspect of varying delays among the immersive participants can potentially cause problems during dynamic interactions and affect their QoE. Answers to questions such as what type of problems can be caused and how the participants handle them depend on the application domain of the immersive environments. To study the QoE and validate (with usability studies) the collaborative, immersive environment, a tele-rehabilitation application will be deployed in multiple cities: Berkeley, California; 2 sites in Dallas, Texas; and Urbana-Champaign, Illinois. Intellectual Merits of this project are (i) The resource adaptation framework for streaming multi-source, multi-destination, multi-rate, multi-modal data incorporates supervisory hybrid control theory based fine-grained resource management, multi-modal coarse-grained management, and a multi-modal multicasting approach. (ii) Graphics Processing Unit (GPU)-based 3D reconstruction and compression algorithms. These algorithms facilitate reconstruction of 3D data points based on 3D camera array data and compress them at a faster pace than their CPU-based counterparts. (iii) GPU-based rendering algorithm of 3D data on the receiver side. This algorithm will handle potential data loss in 3D camera data streams using skeletal information from BSN data streams. (iv) Identification and measurement of Quality of Experience (QoE) metrics and using those metrics to derive Quality of Service (QoS) parameters. The derived QoS parameters will then help the resource adaptation framework to modify its decisions at run-time. This project aims to have transformative aspects in the new set of algorithms that exploits multi-modality while incorporating a feedback based on Quality of Experience for functions such as streaming, 3D reconstruction, and rendering.Broader Impacts: This project promises significant impact in the fields of education and pervasive health care by providing augmented abilities to carry out intricate programs such as tele-rehabilitation with increased correctness and flexibility. This can also lead to improved productivity in the society considering the ability of health-care professionals to potentially handle a larger population (in remote places) as well as considering the possibility of the affected persons to become independent and productive faster. The project also ensures the results from the proposed research will be incorporated into the courses being taught. 3 women PhD students and 6 under-graduate students (2 are minority students) already working with the investigators of this project. Serious efforts will be undertaken to continue their involvement in this project. Apart from refereed conference and journal publications, the developed software, collected data, and research results will be shared with other researchers through a dedicated website (after ensuring satisfaction of HIPAA regulations).

提供一个兼具沉浸感和互动性的环境是一项艰巨的研究挑战。确保在使用安装在地理分散的城市中的这些环境时获得合理的体验质量 (QoE) 甚至是一项艰巨的挑战。这个项目考虑的是一个协作、沉浸式、交互式的环境，不仅支持参与者的 3D 渲染？视频以及其他形式，例如身体传感器网络 (BSN) 数据，可以提供有关人的身体运动（以及生理数据）的高精度数据。在创建这一环境时，需要考虑阻碍承载沉浸式交互信息的数据流的各种瓶颈：重建延迟、所需的超高吞吐量、丢包和渲染延迟。该项目的主要目的是通过执行研究任务来设计和开发具有更高帧速率和帧质量的协作、多模式沉浸式环境，这些任务可以利用来自其他模式的信息并处理这些瓶颈。环境中，参与者可以在本地渲染的 3D 视图中看到自己，也可以看到远程环境中的参与者。由于本地渲染延迟要小得多，因此与遭受通信延迟和数据包丢失的远程参与者的渲染相比，参与者可以更早、更流畅地看到自己。沉浸式参与者之间不同延迟的这一方面可能会在动态交互过程中引起问题并影响他们的 QoE。诸如可能引起什么类型的问题以及参与者如何处理这些问题等问题的答案取决于沉浸式环境的应用领域。为了研究 QoE 并验证（通过可用性研究）协作、沉浸式环境，远程康复应用程序将在多个城市部署：加利福尼亚州伯克利；德克萨斯州达拉斯有 2 个站点；和伊利诺伊州厄巴纳-尚佩恩。该项目的智力优点是（i）流式多源、多目的地、多速率、多模态数据的资源适应框架结合了基于监督混合控制理论的细粒度资源管理、多模态粗粒度管理，以及多模式多播方法。 (ii) 基于图形处理单元 (GPU) 的 3D 重建和压缩算法。这些算法有助于基于 3D 相机阵列数据重建 3D 数据点，并以比基于 CPU 的同行更快的速度压缩它们。 (iii)接收端基于GPU的3D数据渲染算法。该算法将使用 BSN 数据流中的骨架信息来处理 3D 相机数据流中潜在的数据丢失。 (iv) 识别和测量体验质量 (QoE) 指标，并使用这些指标来导出服务质量 (QoS) 参数。然后，导出的 QoS 参数将帮助资源适应框架在运行时修改其决策。该项目旨在在利用多模态的新算法中实现变革性的方面，同时结合基于体验质量的反馈来实现流、3D 重建和渲染等功能。 更广泛的影响：该项目有望在该领域产生重大影响通过提供增强的能力来执行复杂的计划，例如具有更高正确性和灵活性的远程康复，从而促进教育和普及医疗保健的发展。考虑到医疗保健专业人员有可能处理更多人口（在偏远地区），以及考虑到受影响的人更快地独立和生产力的可能性，这也可以提高社会生产力。该项目还确保拟议研究的结果将纳入所教授的课程中。 3 名女博士生和 6 名本科生（2 名少数民族学生）已与该项目的研究人员一起工作。我们将认真努力继续参与该项目。除了经过审阅的会议和期刊出版物外，开发的软件、收集的数据和研究结果将通过专门的网站与其他研究人员共享（在确保满足 HIPAA 法规后）。