Monotonic Neural Network based Rate-Distortion Modeling for H.266/VVC

Employing neural networks (NN) for rate-distortion (R-D) modeling in image/video coding has received growing attention. Typically, NNs are used to capture the R-D relationship through the rate-quantizer (R-Q) and distortion-quantizer (D-Q) models in various works. However, existing NN-based schemes do not consider the monotonicity of the R-D relationship, resulting in inferior distortion predictions. We design a monotonic neural network-based scheme to tackle this issue. We first adopt linear scaling to ground truth to make the R-Q and D-Q relationship always monotonic increasing. We then train an NN to learn the mapping from the latent features and quantizer to the rate/distortion while enforcing the derivative of NN predicted R/D w.r.t the QP is strictly positive. As such, the monotonic R-D relationship can be maintained. Simulation results on the Vimeo dataset with H.266/VVC show that the proposed approach is better than the state-of-the-art NN-based scheme in prediction accuracy and maintaining monotonicity.