Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Deep Inter Prediction with Error-Corrected Auto-Regressive Network for Video Coding

Deep Inter Prediction with Error-Corrected Auto-Regressive Network for Video Coding Modern codecs remove temporal redundancy of a video via inter prediction, i.e., searching previously coded frames for similar blocks and storing motion vectors to save bit-rates. However, existing codecs adopt block-level motion estimation, where a block is regressed by reference blocks linearly and is doomed to fail to deal with non-linear motions. In this article, we generate virtual reference frames (VRFs) with previously reconstructed frames via deep networks to offer an additional candidate, which is not constrained to linear motion structure and further significantly improves coding efficiency. More specifically, we propose a novel deep Auto-Regressive Moving-Average (ARMA) model, Error-Corrected Auto-Regressive Network (ECAR-Net), equipped with the powers of the conventional statistic ARMA models and deep networks jointly for reference frame prediction. Similar to conventional ARMA models, the ECAR-Net consists of two stages: Auto-Regression (AR) stage and Error-Correction (EC) stage, where the first part predicts the signal at the current time-step based on previously reconstructed frames, while the second one compensates for the output of the AR stage to obtain finer details. Different from the statistic AR models only focusing on short-term temporal dependency, the AR model of our ECAR-Net is further injected with the long-term dynamics mechanism, where long temporal information is utilized to help predict motions more accurately. Furthermore, ECAR-Net works in a configuration-adaptive way, i.e., using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios. With the well-designed network, our method surpasses HEVC on average 5.0% and 6.6% BD-rate saving for the luma component under the Low Delay B and Random Access configurations and also obtains on average 1.54% BD-rate saving over VVC. Furthermore, ECAR-Net works in a configuration-adaptive way, i.e., using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Multimedia Computing Communications and Applications (TOMCCAP) Association for Computing Machinery

Deep Inter Prediction with Error-Corrected Auto-Regressive Network for Video Coding

Download PDF
22 pages

Loading next page...
 
/lp/association-for-computing-machinery/deep-inter-prediction-with-error-corrected-auto-regressive-network-for-nQeq2HozV0
Publisher
Association for Computing Machinery
Copyright
Copyright © 2023 Association for Computing Machinery.
ISSN
1551-6857
eISSN
1551-6865
DOI
10.1145/3528173
Publisher site
See Article on Publisher Site

Abstract

Modern codecs remove temporal redundancy of a video via inter prediction, i.e., searching previously coded frames for similar blocks and storing motion vectors to save bit-rates. However, existing codecs adopt block-level motion estimation, where a block is regressed by reference blocks linearly and is doomed to fail to deal with non-linear motions. In this article, we generate virtual reference frames (VRFs) with previously reconstructed frames via deep networks to offer an additional candidate, which is not constrained to linear motion structure and further significantly improves coding efficiency. More specifically, we propose a novel deep Auto-Regressive Moving-Average (ARMA) model, Error-Corrected Auto-Regressive Network (ECAR-Net), equipped with the powers of the conventional statistic ARMA models and deep networks jointly for reference frame prediction. Similar to conventional ARMA models, the ECAR-Net consists of two stages: Auto-Regression (AR) stage and Error-Correction (EC) stage, where the first part predicts the signal at the current time-step based on previously reconstructed frames, while the second one compensates for the output of the AR stage to obtain finer details. Different from the statistic AR models only focusing on short-term temporal dependency, the AR model of our ECAR-Net is further injected with the long-term dynamics mechanism, where long temporal information is utilized to help predict motions more accurately. Furthermore, ECAR-Net works in a configuration-adaptive way, i.e., using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios. With the well-designed network, our method surpasses HEVC on average 5.0% and 6.6% BD-rate saving for the luma component under the Low Delay B and Random Access configurations and also obtains on average 1.54% BD-rate saving over VVC. Furthermore, ECAR-Net works in a configuration-adaptive way, i.e., using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios.

Journal

ACM Transactions on Multimedia Computing Communications and Applications (TOMCCAP) – Association for Computing Machinery

Published: Jan 23, 2023

Keywords: High Efficient Video Coding (HEVC)

References

  • Overview of the H
    [,
  • Overview of the high efficiency video coding (HEVC) standard
    [,
  • Versatile video coding (draft 9)
    [,
  • Dmcnn: Dual-domain multi-scale convolutional neural network for compression artifacts removal
    [,
  • Image/video restoration via multiplanar autoregressive model and low-rank optimization
    [,
  • Isophote-constrained autoregressive model with adaptive window extension for image interpolation
    [,
  • Adaptive general scale interpolation based on weighted autoregressive models
    [,
  • Augmented coarse-to-fine video frame synthesis with semantic loss
    [,
  • Enhanced intra prediction with recurrent neural network in video coding
    [,
  • Progressive spatial recurrent neural network for intra prediction
    [,
  • Optimized spatial recurrent network for intra prediction in video coding
    [,
  • A group variational transformation neural network for fractional interpolation of video coding
    [,
  • One-for-all: Grouped variation network-based fractional interpolation in video coding
    [,
  • Partition tree guided progressive rethinking network for in-loop filtering of HEVC
    [,
  • Enhanced motion-compensated video coding with deep virtual reference frame generation
    [,
  • Video frame interpolation via adaptive separable convolution
    [,
  • Deep frame prediction for video coding
    [,
  • Deep reference generation with multi-domain hierarchical constraints for inter prediction
    [,
  • HEVC inter coding using deep recurrent neural networks and artificial reference pictures
    [,
  • Deep predictive coding networks for video prediction and unsupervised learning
    [,
  • Deep generative image models using a laplacian pyramid of adversarial networks
    [,
  • Generative adversarial network-based frame extrapolation for video coding
    [,
  • FlowNet 2
    [,
  • PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume
    [,
  • Recurrent spatial pyramid CNN for optical flow estimation
    [,
  • Phase-based frame interpolation for video
    [,
  • Video frame interpolation via adaptive convolution
    [,
  • Video frame synthesis using deep voxel flow
    [,
  • Sdc-net: Video prediction using spatially-displaced convolution
    [,
  • Fully connected network-based intra prediction for image coding
    [,
  • DMCNN: Dual-domain multi-scale convolutional neural network for compression artifacts removal
    [,
  • Reference-guided deep super-resolution via manifold localized external compensation
    [,
  • Multi-modal/multi-scale convolutional neural network based in-loop filter design for next generation video codec
    [,
  • Content-aware convolutional neural network for in-loop filtering in high efficiency video coding
    [,
  • Neural network based inter prediction for HEVC
    [,
  • Enhanced ctu-level inter prediction with deep frame rate up-conversion for high efficiency video coding
    [,
  • Long short-term memory
    [,
  • Deep network-based frame extrapolation with reference frame alignment
    [,
  • Similarity modulated block estimation for image interpolation
    [,
  • Marlow: A joint multiplanar autoregressive and low-rank approach for image completion
    [,
  • Convolutional LSTM network: A machine learning approach for precipitation nowcasting
    [,
  • Optical flow estimation using a spatial pyramid network
    [,
  • Learning enriched features for real image restoration and enhancement
    [,
  • Video enhancement with task-oriented flow
    [,
  • Adam: A method for stochastic optimization
    [,
  • Common test conditions and software reference configurations
    [,
  • Calculation of average PSNR differences between RD-curves
    [,
  • Memory-augmented auto-regressive network for frame recurrent inter prediction
    [,
  • DVC: An end-to-end deep video compression framework
    [,