NetSE:Large:Collaborative Research: Exploiting Multi-modality for Tele-Immersion

NetSE：大型：协作研究：利用多模态实现远程沉浸

• 批准号: 1012975
• 负责人: Balakrishnan Prabhakaran
• 金额: $ 203.38万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-10-01 至 2018-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1012975&HistoricalAwards=false
• 关键词: NetSE; Large; Collaborative; Research; Exploiting

Providing an environment that offers both immersion and interaction is a tough research challenge. Ensuring a reasonable Quality of Experience (QoE) in using these environments installed in geographically distributed cities is even a tougher challenge. This project considers a collaborative, immersive, and interactive environment that not only supports 3D rendering of the participants? video but also other modalities such as Body Sensor Network (BSN) data that can offer highly precise data about a person?s physical movements (as well as physiological data). While creating this environment, one needs to consider the various bottlenecks that choke the data streams carrying the immersive and interactive information: reconstruction delay, ultra-high throughput needed, packet loss, and rendering delays. The main aim of this project is to design and develop collaborative, multi-modal immersive environments with higher frame rates and frame quality by carrying out research tasks that can take advantage of information from other modalities and handle these bottlenecks.In a typical tele-immersive environment, participants can see themselves in the locally rendered 3D view and see participants in the remote environments as well. Since the local rendering delays are much smaller, participants can see themselves earlier and in a more smooth fashion compared to the rendering of remote participants that suffers from communication delays and packet losses. This aspect of varying delays among the immersive participants can potentially cause problems during dynamic interactions and affect their QoE. Answers to questions such as what type of problems can be caused and how the participants handle them depend on the application domain of the immersive environments. To study the QoE and validate (with usability studies) the collaborative, immersive environment, a tele-rehabilitation application will be deployed in multiple cities: Berkeley, California; 2 sites in Dallas, Texas; and Urbana-Champaign, Illinois. Intellectual Merits of this project are (i) The resource adaptation framework for streaming multi-source, multi-destination, multi-rate, multi-modal data incorporates supervisory hybrid control theory based fine-grained resource management, multi-modal coarse-grained management, and a multi-modal multicasting approach. (ii) Graphics Processing Unit (GPU)-based 3D reconstruction and compression algorithms. These algorithms facilitate reconstruction of 3D data points based on 3D camera array data and compress them at a faster pace than their CPU-based counterparts. (iii) GPU-based rendering algorithm of 3D data on the receiver side. This algorithm will handle potential data loss in 3D camera data streams using skeletal information from BSN data streams. (iv) Identification and measurement of Quality of Experience (QoE) metrics and using those metrics to derive Quality of Service (QoS) parameters. The derived QoS parameters will then help the resource adaptation framework to modify its decisions at run-time. This project aims to have transformative aspects in the new set of algorithms that exploits multi-modality while incorporating a feedback based on Quality of Experience for functions such as streaming, 3D reconstruction, and rendering.Broader Impacts: This project promises significant impact in the fields of education and pervasive health care by providing augmented abilities to carry out intricate programs such as tele-rehabilitation with increased correctness and flexibility. This can also lead to improved productivity in the society considering the ability of health-care professionals to potentially handle a larger population (in remote places) as well as considering the possibility of the affected persons to become independent and productive faster. The project also ensures the results from the proposed research will be incorporated into the courses being taught. 3 women PhD students and 6 under-graduate students (2 are minority students) already working with the investigators of this project. Serious efforts will be undertaken to continue their involvement in this project. Apart from refereed conference and journal publications, the developed software, collected data, and research results will be shared with other researchers through a dedicated website (after ensuring satisfaction of HIPAA regulations).

提供既有沉浸式和互动的环境是一项艰巨的研究挑战。在使用地理位置分布的城市中安装的这些环境中，确保合理的经验（QOE）是一个更艰巨的挑战。该项目考虑了一个协作，沉浸式和互动的环境，不仅支持参与者的3D渲染？视频以及其他模式，例如身体传感器网络（BSN）数据，这些数据可以提供有关人的身体运动（以及生理数据）的高度精确数据。在创建此环境的同时，需要考虑扼杀带有沉浸式和交互信息的数据流的各种瓶颈：重建延迟，所需的超高吞吐量，数据包丢失和渲染延迟。该项目的主要目的是通过执行可以利用其他方式的信息并处理这些瓶颈的研究任务来设计和开发具有更高框架速率和框架质量的协作，多模式的沉浸式环境。由于本地渲染延迟要小得多，因此与远程参与者的渲染相比，参与者可以更早地看到自己，并且以沟通延迟和数据包损失的效果更加顺利。身临其境参与者之间不同延误的这一方面可能会在动态互动过程中引起问题并影响其QOE。回答诸如可能引起哪种类型问题以及参与者如何处理问题的问题取决于身临其境环境的应用领域。为了研究Qoe并验证（有可用性研究）协作，沉浸式环境，将在多个城市部署远程审查的应用：加利福尼亚州伯克利；德克萨斯州达拉斯的2个地点；和伊利诺伊州的Urbana-Champaign。该项目的智力优点是（i）流式传输多源，多末端，多速率的多模式数据的资源适应框架，包括基于监督混合控制理论，基于高级粒度资源管理，多模式的粗粒度管理，多模式的多模式多模式多模式的多模式播种方法。 （ii）基于图形处理单元（GPU）的3D重建和压缩算法。这些算法促进了基于3D摄像机数组数据的3D数据点的重建，并以比基于CPU的配料更快地压缩它们。 （iii）基于GPU的接收器侧3D数据的渲染算法。该算法将使用BSN数据流中的骨骼信息处理3D摄像机数据流中的潜在数据丢失。 （iv）识别和测量经验质量（QOE）指标，并使用这些指标来得出服务质量（QOS）参数。然后，派生的QoS参数将帮助资源适应框架在运行时修改其决策。该项目旨在在新的算法集中具有变革性的方面，这些算法利用了多模式，同时根据经验的反馈来纳入流式传输，3D重建和渲染的功能质量。BRODERING的影响：该项目有望在教育领域和范围内提高能力以及携带更多的传播能力，从而在教育领域中产生重大影响。考虑到卫生保健专业人员有可能处理更大的人口（在偏远地区）的能力，并考虑受影响的人更快地独立和生产力的可能性，这也可能会提高社会的生产率。该项目还确保了拟议研究的结果将纳入所教的课程中。已经与该项目的调查员一起工作的3名女博士生和6名本科生（2名是少数族裔学生）。将采取严肃的努力继续参与该项目。除了被裁判的会议和期刊出版物外，开发的软件，收集的数据以及研究结果将通过专用网站与其他研究人员共享（确保对HIPAA法规满意之后）。