End-to-End Neural Speech Coding for Real-Time Communications

Deep-learning based methods have shown their advantages in audio coding over traditional ones but limited attention has been paid on real-time communications (RTC). This paper proposes the TFNet, an end-to-end neural speech codec with low latency for RTC. It takes an encoder-temporal filtering-decoder paradigm that has seldom been investigated in audio coding. An interleaved structure is proposed for temporal filtering to capture both short-term and long-term temporal dependencies. Furthermore, with end-to-end optimization, the TFNet is jointly optimized with speech enhancement and packet loss concealment, yielding a one-for-all network for three tasks. Both subjective and objective results demonstrate the efficiency of the proposed TFNet.

基于深度学习的方法在音频编码方面相较于传统方法已展现出优势，但在实时通信（RTC）领域却鲜受关注。本文提出了TFNet，这是一种面向实时通信的低延迟端到端神经语音编解码器。它采用了编码器 - 时间滤波 - 解码器范式，而这种范式在音频编码中鲜有人研究。为时间滤波提出了一种交错结构，以捕捉短期和长期的时间依赖关系。此外，通过端到端优化，TFNet与语音增强和丢包隐藏功能进行联合优化，形成了一个可兼顾这三项任务的通用网络。主观和客观结果均证明了所提出的TFNet的有效性。