ERI: Generative Adversarial Networks for Video Coding

ERI:用于视频编码的生成对抗网络

基本信息

  • 批准号:
    2138635
  • 负责人:
  • 金额:
    $ 19.62万
  • 依托单位:
  • 依托单位国家:
    美国
  • 项目类别:
    Standard Grant
  • 财政年份:
    2022
  • 资助国家:
    美国
  • 起止时间:
    2022-02-01 至 2025-01-31
  • 项目状态:
    未结题

项目摘要

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Video coding is an important technology that compresses video signals to save transmission bandwidth and to provide Internet users with visually pleasing decoded videos. Inspired by recent breakthroughs in deep learning, convolutional neural networks have been increasingly exploited into video coding algorithms to provide significant coding gains compared to conventional approaches. Nevertheless, existing convolutional neural network-based video coding schemes tend to generate blurry decoded images which are inconsistent with human perception, and the high computational complexity of these schemes hinders their deployment on power-constrained and computation resource-limited devices, such as smart phones and tablets. Recently, the generative adversarial network demonstrated its capability of decoding sharp and photo-realistic images at low bit rates, but little research has investigated its potential for video compression. This project will develop generative adversarial network-based video coding systems to enhance the coding efficiency, meanwhile providing decoded videos with high perceptual quality. The project will also investigate low-complexity algorithms to reduce the power consumption and to accelerate the inference speed of the proposed video coding systems so that they are suitable for mobile and low-latency applications. The success of the project is expected to accelerate the economic growth of streaming video services to benefit people’s daily professional and entertainment activities. It will advance surveillance video services to enhance public safety in places such as airport, offices, highway, and road intersections. The research activities of the project will provide opportunities to train graduate and undergraduate students including minority and under-represented groups through theses research, senior design projects, as well as machine learning and artificial intelligence courses. The research results of the project will be showcased in a summer engineering seminar program to motivate high school students to pursue science and engineering majors in college.This project will address two problems: (1) How to leverage temporal correlations among video frames and explore scene dynamics in a generative adversarial network-based video coding architecture? Two approaches are proposed: a hierarchical predictive coding approach, and a spatial-temporal coding architecture based on 3-dimensional convolution. Since most existing generative adversarial network models are for still image compression, the success of this research will open the door to generative adversarial network-based coding systems for video coding professionals. (2) How to reduce the computational complexity of deep video coding networks? Despite the performance benefits of deep learning-based video coding tools, few of them are currently being adopted in real-world scenarios. This is due to the high computational complexity, slow inference speed and the large graphic processing unit memory requirements associated with deep network computation. To address this problem, the proposed research will develop algorithms to reduce the complexity, model size and model parameters of deep learning-based video coding models via separable convolution operations. The research results will accelerate the deployment of deep video coding models in real-world applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
该奖项是根据2021年《美国救援计划法》的全部或部分资助(公共法律117-2)。VIDEO编码是一项重要技术,可压缩视频信号以节省传输带宽并为互联网用户提供视觉上令人满意的解码视频。受深度学习的最新突破的启发,卷积神经网络已越来越多地探索到视频编码算法中,与传统方法相比,可以提供显着的编码收益。然而,现有的基于卷积神经网络的视频编码方案倾向于产生模糊的解码图像,这些图像与人类的感知不一致,并且这些方案的高计算复杂性阻碍了它们在功率受限和计算资源限制的设备上的部署,例如智能手机和平板电脑。最近,通用的对抗网络证明了其以低比特率以低比率解码锋利和照片现实图像的能力,但是很少的研究研究了其视频压缩的潜力。该项目将开发一般的基于网络的视频编码系统,以提高编码效率,同时提供具有高感知质量的解码视频。该项目还将研究低复杂性算法,以减少功耗并加速提议的视频编码系统的推理速度,以便它们适合移动和低延迟应用程序。预计该项目的成功将加速流媒体视频服务的经济增长,从而使人们的日常专业和娱乐活动受益。它将推进监视视频服务,以增强机场,办公室,高速公路和道路交叉路口等地方的公共安全。该项目的研究活动将通过这些研究,高级设计项目以及机器学习和人工智能课程来培训培训毕业生和本科生,包括少数民族和代表性不足的团体。该项目的研究结果将在夏季工程开创性计划中展示,以激励高中生在大学中攻读科学和工程专业的专业。该项目将解决两个问题:(1)如何利用视频框架之间的临时相关性并探索基于通用的对手网络基于网络的视频编码体系结构中的场景动态?提出了两种方法:一种层次预测编码方法,以及基于三维卷积的时空编码结构。由于大多数现有的通用对抗网络模型都是用于静止图像压缩的,因此这项研究的成功将为视频编码专业人员的基于对抗性网络的通用网络编码系统打开大门。 (2)如何降低深视频编码网络的计算复杂性?尽管基于深度学习的视频编码工具具有性能好处,但在现实世界中,其中很少有人被采用。这是由于高计算复杂性,缓慢的推理速度和与深网计算相关的大图形处理单元内存需求。为了解决这个问题,拟议的研究将开发算法,以通过单独的卷积操作来减少基于深度学习的视频编码模型的复杂性,模型大小和模型参数。研究结果将加速现实世界应用中深层视频编码模型的部署。该奖项反映了NSF的法定任务,并使用基金会的知识分子优点和更广泛的影响审查标准,认为通过评估被认为是宝贵的支持。

项目成果

期刊论文数量(6)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
A generative adversarial network for video compression
  • DOI:
    10.1117/12.2618714
  • 发表时间:
    2022-05
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Pengli Du;Ying Liu;Nam Ling;Lingzhi Liu;Yongxiong Ren;M. Hsu
  • 通讯作者:
    Pengli Du;Ying Liu;Nam Ling;Lingzhi Liu;Yongxiong Ren;M. Hsu
Learned image compression with transformers
  • DOI:
    10.1117/12.2656516
  • 发表时间:
    2023-06
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Tianma Shen;Y. Liu
  • 通讯作者:
    Tianma Shen;Y. Liu
A Survey of Efficient Deep Learning Models for Moving Object Segmentation
用于运动物体分割的高效深度学习模型综述
Side Information Driven Image Coding for Machines
  • DOI:
    10.1109/pcs56426.2022.10018039
  • 发表时间:
    2022-12
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Zhongpeng Zhang;Y. Liu
  • 通讯作者:
    Zhongpeng Zhang;Y. Liu
Generative Video Compression with a Transformer-Based Discriminator
  • DOI:
    10.1109/pcs56426.2022.10018030
  • 发表时间:
    2022-12
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Pengli Du;Y. Liu;Nam Ling;Yongxiong Ren;Lingzhi Liu
  • 通讯作者:
    Pengli Du;Y. Liu;Nam Ling;Yongxiong Ren;Lingzhi Liu
{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

Ying Liu其他文献

T2K実験新型前置検出器を用いた電子ニュートリノ事象選別のための反応点再構成手法の開発
T2K实验中新型前端探测器电子中微子事件选择反应点重建方法的发展
  • DOI:
  • 发表时间:
    2023
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Yukie Kashima;Ayako Suzuki;Ying Liu;Masahito Hosokawa;Hiroko Matsunaga;Masataka Shirai;Kohji Arikawa;Sumio Sugano;Takashi Kohno;Haruko Takeyama;Katsuya Tsuchihara & Yutaka Suzuki;瀧口優,豊田晴義;I. Ushijima;小林北斗
  • 通讯作者:
    小林北斗
Pilot Study on Cytothesis for rabbit ocular surface defects using peripheral blood autologous mononuclear cells
外周血自体单个核细胞细胞合成修复兔眼表缺损的初步研究
  • DOI:
  • 发表时间:
    2020
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Hening Zhang;Qijiong Li;M. Zhang;Minglei Zhao;Mian Huang;Weihua Li;Wencong Wang;Bikun Xian;Ying Liu;Zhiquan Li;Yaojue Xie;Xiulan Zhang;Zhichong Wang;Bing Huang
  • 通讯作者:
    Bing Huang
Optimal analysis for thermal conductivity variation of EVA/SCF composites prepared by spatial confining forced network assembly
空间约束强制网络组装EVA/SCF复合材料热导率变化的优化分析
  • DOI:
    10.1016/j.mtcomm.2020.101206
  • 发表时间:
    2020-12
  • 期刊:
  • 影响因子:
    3.8
  • 作者:
    Shuhui Wang;Ying Liu;Yang Guo;Yuan Lu;Yao Huang;Hong Xu;Daming Wu;Jingyao Sun
  • 通讯作者:
    Jingyao Sun
Sticking behaviour and mechanism of iron ore pellets in COREX pre-reduction shaft furnace
铁矿球团矿在COREX预还原竖炉中的粘着行为及机理
  • DOI:
    10.1080/03019233.2017.1361079
  • 发表时间:
    2019-02
  • 期刊:
  • 影响因子:
    2.1
  • 作者:
    Zhan-xia Di;Zheng-yi Li;Ru-fei Wei;Ying Liu;Qing-min Meng;Tie-jun Chun;Hong-ming Long;Jia-xin Li;Ping Wang
  • 通讯作者:
    Ping Wang
Cross-sectional Exploration of the Relationship Between Glutamate Abnormalities and Tic Disorder Severity Using Proton Magnetic Resonance Spectroscopy
使用质子磁共振波谱法横断面探索谷氨酸异常与抽动障碍严重程度之间的关系
  • DOI:
    10.1007/s43657-022-00064-z
  • 发表时间:
    2022
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Juanjuan Hao;Xin Zhang;Ying Liu;Zhongyang Zhang;Keyu Jiang;Xiao-Yong Zhang;Min Wu
  • 通讯作者:
    Min Wu

Ying Liu的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

{{ truncateString('Ying Liu', 18)}}的其他基金

EAGER: Resolving the issue of pairing symmetry in Sr2RuO4
EAGER:解决 Sr2RuO4 中的配对对称性问题
  • 批准号:
    2312899
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
I-CORPS: Scalable Production of Polymeric Nanoparticles Encapsulating Hydrophobic Compounds
I-CORPS:封装疏水性化合物的聚合物纳米颗粒的可规模化生产
  • 批准号:
    1566113
  • 财政年份:
    2015
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
CAREER: Understanding Nanoprecipitation - Scalable Production of Polymeric Nanoparticles Encapsulating Hydrophobic Compounds
职业:了解纳米沉淀 - 封装疏水性化合物的聚合物纳米颗粒的规模化生产
  • 批准号:
    1350731
  • 财政年份:
    2014
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Toroidal-spiral particles (TSPs) for co-delivery of multiple compounds of different sizes
用于共同递送多种不同尺寸化合物的环形螺旋颗粒 (TSP)
  • 批准号:
    1404884
  • 财政年份:
    2014
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
EAGER: Preliminary Study on Novel self-assembled Toroidal-Spiral MicroParticles (TSMPs) for sustained release of therapeutic proteins and peptides: theory and experiments
EAGER:用于持续释放治疗性蛋白质和肽的新型自组装环形螺旋微粒(TSMP)的初步研究:理论和实验
  • 批准号:
    1039531
  • 财政年份:
    2010
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Materials World Network: Novel Physical Phenomena in Unusual Mesoscopic Superconductors
材料世界网络:异常介观超导体中的新物理现象
  • 批准号:
    0908700
  • 财政年份:
    2009
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
US-France Cooperative Research: Search for Edge Currents and Domain Walls in SrRu0
美法合作研究:寻找SrRu0中的边缘电流和畴壁
  • 批准号:
    0340779
  • 财政年份:
    2004
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Experimental Studies of Nanoscopic Superconductors: Half-flux Quantum, Metallic State of Cooper Pairs, and the Berry's Phase
纳米超导体的实验研究:半通量量子、库珀对金属态和贝里相
  • 批准号:
    0202534
  • 财政年份:
    2002
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Continuing Grant
Determination of the Exact Symmetry of the Pairing State in Sr2RuO4
Sr2RuO4 中配对态精确对称性的测定
  • 批准号:
    9974327
  • 财政年份:
    1999
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Continuing Grant
CAREER: Mesoscopic Physics of Disordered Superconductors: An Arena for Research and Education
职业:无序超导体的介观物理:研究和教育的舞台
  • 批准号:
    9702661
  • 财政年份:
    1997
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Continuing Grant

相似国自然基金

面向物体触觉属性感知的视触跨模态生成方法研究
  • 批准号:
    62303259
  • 批准年份:
    2023
  • 资助金额:
    10 万元
  • 项目类别:
    青年科学基金项目
VSMC机械感受器TRPM7调控H3S10p/NOTCH3促进冠状动脉侧支生成的作用与机制研究
  • 批准号:
    82300366
  • 批准年份:
    2023
  • 资助金额:
    30 万元
  • 项目类别:
    青年科学基金项目
基于流模型的高安全图像生成式隐写研究
  • 批准号:
    62372125
  • 批准年份:
    2023
  • 资助金额:
    50 万元
  • 项目类别:
    面上项目
核燃料包壳用FeCrAl合金中非金属夹杂物生成、演变及其调控机理
  • 批准号:
    52374341
  • 批准年份:
    2023
  • 资助金额:
    50 万元
  • 项目类别:
    面上项目
污水再生处理及地下储存体系曝气孔口气泡微细化调控生成方法与效能
  • 批准号:
    52370032
  • 批准年份:
    2023
  • 资助金额:
    50 万元
  • 项目类别:
    面上项目

相似海外基金

SBIR Phase I: High Fidelity Climate Simulation Powered by Generative Adversarial Networks
SBIR 第一阶段:由生成对抗网络提供支持的高保真气候模拟
  • 批准号:
    2335370
  • 财政年份:
    2024
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Pure transformer encoder-based generative adversarial networks for molecular generation
用于分子生成的基于纯变压器编码器的生成对抗网络
  • 批准号:
    23KF0063
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Grant-in-Aid for JSPS Fellows
Information-Theoretic Surprise-Driven Approach to Enhance Decision Making in Healthcare
信息论惊喜驱动方法增强医疗保健决策
  • 批准号:
    10575550
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
Geles: A Novel Imaging Informatics System for Generalizable Lesion Identification in Neuroendocrine Tumors
Geles:一种用于神经内分泌肿瘤普遍病变识别的新型影像信息学系统
  • 批准号:
    10740578
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
HomePal: Developing a Smart Speaker-Based System for In-Home Loneliness Assessment for Older Adults
HomePal:开发基于智能扬声器的系统,用于老年人的家庭孤独评估
  • 批准号:
    10725229
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了