CAREER: Machine-centered Cyberinfrastructure for Panoramic Video Analytics in Science and Engineering Monitoring

职业：科学和工程监控中用于全景视频分析的以机器为中心的网络基础设施

• 批准号: 2144764
• 负责人: Zhisheng Yan
• 金额: $ 50万
• 依托单位: George Mason University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-15 至 2027-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2144764&HistoricalAwards=false
• 关键词: CAREER; Machine; centered; Cyberinfrastructure; Panoramic

Video analytics plays a pivotal role in science and engineering monitoring. Monitoring videos captured by remote cameras are typically live streamed to servers for analysis because of the limited computational capabilities of camera devices. From wildlife tracking and coastline event detection to airport suspect recognition and victim search in disaster response, such automated video analytics systems have been deployed widely to assist human operators. The recent advancement of 360 degree cameras enables a new paradigm of panoramic video analytics that can cover the 360 degree surroundings of a monitoring site and can address the errors in and missing analysis abilities of traditional 2D video analytics. However, realizing this vision requires live streaming massive panoramic video data to servers for online analytics, which cannot be supported by the current cyberinfrastructure (CI). The mismatch between the 360 degree video bit rate and available network bandwidth can cause lagging or failed analysis, diminishing the benefits of panoramic video analytics. This project will create a framework of video compression, streaming, and recovery for achieving the vision of panoramic video analytics in science and engineering monitoring. The new CI will allow scientists and engineers to conduct online panoramic video analytics and enable innovative applications that are otherwise unattainable. The research outcomes will support the development of a remote learning tool for imaging analytics, course curriculum and undergraduate research in media computing, and educational videos for public outreach.This project investigates a machine centered video computing framework in order to enable online panoramic video analytics. Unlike traditional human centered video frameworks where pixels are processed to preserve extensive aesthetic details for human viewing, the proposed CI compresses, streams, and recovers feature points for machine analytics. Because of this fundamental change, the proposed framework is able to greatly outperform legacy video CIs and support panoramic video analytics. To this end, a deep learning based 360 degree video codec will be built to distill the spatiotemporal characteristics of video features and optimize both compression ratio and analytics accuracy. Second, an adaptive 360 degree video bitrate streaming system will be designed to ensure continuous delivery of full 360 degree video frames by prioritizing regions of interest preferred by machines. Third, a 360 degree video recovery scheme will be developed to restore noisy and delayed video data while considering the time constraints in the online analytics models. Finally, interdisciplinary collaboration will be done with application area scientists and engineers to carry out the project plans for evaluation and validation of the panoramic video framework on real world problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

视频分析在科学和工程监控中发挥着关键作用。由于摄像头设备的计算能力有限，远程摄像头捕获的监控视频通常会实时传输到服务器进行分析。从野生动物跟踪和海岸线事件检测到机场嫌疑人识别和灾难响应中的受害者搜索，此类自动视频分析系统已被广泛部署以协助人类操作员。 360度摄像机的最新进展实现了全景视频分析的新范例，可以覆盖监控站点的360度周围环境，并可以解决传统2D视频分析中的错误和分析能力缺失的问题。然而，实现这一愿景需要将海量全景视频数据实时传输到服务器进行在线分析，而当前的网络基础设施（CI）无法支持这一点。 360 度视频比特率与可用网络带宽之间的不匹配可能会导致分析滞后或失败，从而削弱全景视频分析的优势。该项目将创建一个视频压缩、流媒体和恢复框架，以实现科学和工程监控中全景视频分析的愿景。新的 CI 将允许科学家和工程师进行在线全景视频分析，并实现其他方式无法实现的创新应用。研究成果将支持开发用于图像分析的远程学习工具、媒体计算方面的课程和本科生研究以及用于公共宣传的教育视频。该项目研究以机器为中心的视频计算框架，以实现在线全景视频分析。与传统的以人为中心的视频框架不同，传统的以人为中心的视频框架对像素进行处理，以保留大量的美学细节以供人类观看，而所提出的 CI 会压缩、流式传输和恢复特征点以供机器分析。由于这一根本性变化，所提出的框架能够大大优于传统视频 CI，并支持全景视频分析。为此，将构建基于深度学习的360度视频编解码器，以提取视频特征的时空特征并优化压缩比和分析准确性。其次，将设计自适应 360 度视频比特率流系统，通过优先考虑机器首选的感兴趣区域，确保连续交付完整的 360 度视频帧。第三，将开发360度视频恢复方案来恢复嘈杂和延迟的视频数据，同时考虑在线分析模型的时间限制。最后，将与应用领域的科学家和工程师进行跨学科合作，以实施针对现实世界问题评估和验证全景视频框架的项目计划。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准。

项目成果

Enhancing 360 Video Streaming through Salient Content in Head-Mounted Displays

通过头戴式显示器中的显着内容增强 360 度视频流

• DOI: 10.3390/s23084016
• 发表时间: 2023-04-01
• 期刊: Sensors (Basel, Switzerland)
• 作者: Anh Nguyen;Zhisheng Yan
• 通讯作者: Zhisheng Yan

DAO: Dynamic Adaptive Offloading for Video Analytics

DAO：用于视频分析的动态自适应卸载

• DOI: 10.1145/3503161.3548249
• 发表时间: 2022-10-10
• 期刊: Proceedings of the 30th ACM International Conference on Multimedia
• 作者: Taslim Murad;Anh Nguyen;Zhisheng Yan
• 通讯作者: Zhisheng Yan

Context-aware image compression optimization for visual analytics offloading

用于视觉分析卸载的上下文感知图像压缩优化

• DOI: 10.1145/3524273.3528178
• 发表时间: 2022-06
• 期刊: MMSys '22: Proceedings of the 13th ACM Multimedia Systems Conference
• 作者: Chen, Bo;Yan, Zhisheng;Nahrstedt, Klara
• 通讯作者: Nahrstedt, Klara