Paper Info
Title
Weighted Convolutional Motion-Compensated Frame Rate Up-Conversion Using Deep Residual Network
Abstract
Frame rate up-conversion (FRUC) usually suffers from unreliable motion vectors due to the absence of the current frame to be interpolated. In addition, since the majority of video sequences are usually compressed by various coding standards to reduce the data volume, the quality of the generated frames in the FRUC will be further impaired. To address this problem, we proposed two FRUC algorithms based on deep residual network. We first present a deep residual network for the FRUC (DRNFRUC), which consists of feature extraction, feature recursive analysis, and image restoration parts with a skip connection between the input and the output of the network. The proposed DRNFRUC takes the result of an arbitrary existing FRUC method as the input and is able to significantly reduce the edge blurring and blocking artifacts when the motion of the block is violent. In addition, we proposed a deep residual network with weighted convolutional motion compensation (DRNWCMC) for the FRUC, where the convolution operations can be embedded into the motion compensation interpolation (MCI) in any existing MCI-based FRUC method. In DRNWCMC, we first devise two convolutional neural networks corresponding to the forward and backward motion compensated frames, respectively. And then, the adaptive interpolation coefficients for motion compensation are designed as two <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1\times1$ </tex-math></inline-formula> convolutional kernels. Finally, the interpolation result of WCMC is fed into another convolutional neural network to further improve the performance. All the parameters involved in the DRNWCMC are trained simultaneously under the same cost function. The experimental results show that the two proposed algorithms can remarkably improve both the objective and subjective quality of the interpolated frames.
Year
DOI
Venue
2020
10.1109/TCSVT.2018.2885564
IEEE Transactions on Circuits and Systems for Video Technology
Keywords
DocType
Volume
Feature extraction,Motion compensation,Convolutional neural networks,Convolution,Image restoration,Interpolation,Motion estimation
Journal
30
Issue
ISSN
Citations 
1
1051-8215
1
PageRank 
References 
Authors
0.36
12
6
Name
Order
Citations
PageRank
Zhang Y145950.31
Lixin Chen210.36
Chenggang Yan341032.87
Peiwu Qin410.36
Xiangyang Ji553373.14
Qionghai Dai63904215.66