A Survey on Perceptually Optimized Video Coding

Yun ZHANG; Linwei ZHU; Gangyi JIANG; Sam KWONG; C.-C. Jay KUO

doi:10.1145/3571727

A Survey on Perceptually Optimized Video Coding

Yun ZHANG, Linwei ZHU, Gangyi JIANG, Sam KWONG, C.-C. Jay KUO

Research output: Journal Publications › Journal Article (refereed) › peer-review

8 Citations (Scopus)

Abstract

To provide users with more realistic visual experiences, videos are developing in the trends of Ultra High Definition (UHD), High Frame Rate (HFR), High Dynamic Range (HDR), Wide Color Gammut (WCG), and high clarity. However, the data amount of videos increases exponentially, which requires high efficiency video compression for storage and network transmission. Perceptually optimized video coding aims to maximize compression efficiency by exploiting visual redundancies. In this article, we present a broad and systematic survey on perceptually optimized video coding. Firstly, we present problem formulation and framework of the perceptually optimized video coding, which includes visual perception modeling, visual quality assessment, and perceptual video coding optimization. Secondly, recent advances on visual factors, computational perceptual models, and quality assessment models are presented. Thirdly, we review perceptual video coding optimizations from four key aspects, including perceptually optimized bit allocation, rate-distortion optimization, transform and quantization, and filtering and enhancement. In each part, problem formulation, working flow, recent advances, advantages, and challenges are presented. Fourthly, perceptual coding performances of the latest coding standards and tools are experimentally analyzed. Finally, challenging issues and future opportunities are identified.

Original language	English
Article number	3571727
Journal	ACM Computing Surveys
Volume	55
Issue number	12
Early online date	2 Mar 2023
DOIs	https://doi.org/10.1145/3571727
Publication status	Published - 2 Mar 2023
Externally published	Yes

Bibliographical note

Funding Information:
This work was supported in part by the National Natural Science Foundation of China under Grants No. 62172400, No. 61901459, and No. 62271276; in part by the Shenzhen Science and Technology Program under Grant No. JCYJ20200109110410133; in part by the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2022A1515011351; in part by the CAS President’s International Fellowship Initiative (PIFI) under Grant No. 2022VTA0005; in part by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA); and in part by the Hong Kong GRF-RGC General Research Fund under Grants No. 11209819 (CityU 9042816) and No. 11203820 (CityU 9042598).

Publisher Copyright:
© 2023 Association for Computing Machinery.

Keywords

Additional Key Words and PhrasesPerceptual video coding
human visual system
just noticeable difference
quality assessment
rate-distortion optimization
visual attention

Access to Document

10.1145/3571727

Cite this

@article{5558f92720614e48b34cf43ee473e495,

title = "A Survey on Perceptually Optimized Video Coding",

abstract = "To provide users with more realistic visual experiences, videos are developing in the trends of Ultra High Definition (UHD), High Frame Rate (HFR), High Dynamic Range (HDR), Wide Color Gammut (WCG), and high clarity. However, the data amount of videos increases exponentially, which requires high efficiency video compression for storage and network transmission. Perceptually optimized video coding aims to maximize compression efficiency by exploiting visual redundancies. In this article, we present a broad and systematic survey on perceptually optimized video coding. Firstly, we present problem formulation and framework of the perceptually optimized video coding, which includes visual perception modeling, visual quality assessment, and perceptual video coding optimization. Secondly, recent advances on visual factors, computational perceptual models, and quality assessment models are presented. Thirdly, we review perceptual video coding optimizations from four key aspects, including perceptually optimized bit allocation, rate-distortion optimization, transform and quantization, and filtering and enhancement. In each part, problem formulation, working flow, recent advances, advantages, and challenges are presented. Fourthly, perceptual coding performances of the latest coding standards and tools are experimentally analyzed. Finally, challenging issues and future opportunities are identified.",

keywords = "Additional Key Words and PhrasesPerceptual video coding, human visual system, just noticeable difference, quality assessment, rate-distortion optimization, visual attention",

author = "Yun ZHANG and Linwei ZHU and Gangyi JIANG and Sam KWONG and KUO, {C.-C. Jay}",

note = "Funding Information: This work was supported in part by the National Natural Science Foundation of China under Grants No. 62172400, No. 61901459, and No. 62271276; in part by the Shenzhen Science and Technology Program under Grant No. JCYJ20200109110410133; in part by the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2022A1515011351; in part by the CAS President{\textquoteright}s International Fellowship Initiative (PIFI) under Grant No. 2022VTA0005; in part by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA); and in part by the Hong Kong GRF-RGC General Research Fund under Grants No. 11209819 (CityU 9042816) and No. 11203820 (CityU 9042598). Publisher Copyright: {\textcopyright} 2023 Association for Computing Machinery.",

year = "2023",

month = mar,

day = "2",

doi = "10.1145/3571727",

language = "English",

volume = "55",

journal = "ACM Computing Surveys",

issn = "0360-0300",

publisher = "Association for Computing Machinery (ACM)",

number = "12",

}

TY - JOUR

T1 - A Survey on Perceptually Optimized Video Coding

AU - ZHANG, Yun

AU - ZHU, Linwei

AU - JIANG, Gangyi

AU - KWONG, Sam

AU - KUO, C.-C. Jay

N1 - Funding Information: This work was supported in part by the National Natural Science Foundation of China under Grants No. 62172400, No. 61901459, and No. 62271276; in part by the Shenzhen Science and Technology Program under Grant No. JCYJ20200109110410133; in part by the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2022A1515011351; in part by the CAS President’s International Fellowship Initiative (PIFI) under Grant No. 2022VTA0005; in part by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA); and in part by the Hong Kong GRF-RGC General Research Fund under Grants No. 11209819 (CityU 9042816) and No. 11203820 (CityU 9042598). Publisher Copyright: © 2023 Association for Computing Machinery.

PY - 2023/3/2

Y1 - 2023/3/2

N2 - To provide users with more realistic visual experiences, videos are developing in the trends of Ultra High Definition (UHD), High Frame Rate (HFR), High Dynamic Range (HDR), Wide Color Gammut (WCG), and high clarity. However, the data amount of videos increases exponentially, which requires high efficiency video compression for storage and network transmission. Perceptually optimized video coding aims to maximize compression efficiency by exploiting visual redundancies. In this article, we present a broad and systematic survey on perceptually optimized video coding. Firstly, we present problem formulation and framework of the perceptually optimized video coding, which includes visual perception modeling, visual quality assessment, and perceptual video coding optimization. Secondly, recent advances on visual factors, computational perceptual models, and quality assessment models are presented. Thirdly, we review perceptual video coding optimizations from four key aspects, including perceptually optimized bit allocation, rate-distortion optimization, transform and quantization, and filtering and enhancement. In each part, problem formulation, working flow, recent advances, advantages, and challenges are presented. Fourthly, perceptual coding performances of the latest coding standards and tools are experimentally analyzed. Finally, challenging issues and future opportunities are identified.

AB - To provide users with more realistic visual experiences, videos are developing in the trends of Ultra High Definition (UHD), High Frame Rate (HFR), High Dynamic Range (HDR), Wide Color Gammut (WCG), and high clarity. However, the data amount of videos increases exponentially, which requires high efficiency video compression for storage and network transmission. Perceptually optimized video coding aims to maximize compression efficiency by exploiting visual redundancies. In this article, we present a broad and systematic survey on perceptually optimized video coding. Firstly, we present problem formulation and framework of the perceptually optimized video coding, which includes visual perception modeling, visual quality assessment, and perceptual video coding optimization. Secondly, recent advances on visual factors, computational perceptual models, and quality assessment models are presented. Thirdly, we review perceptual video coding optimizations from four key aspects, including perceptually optimized bit allocation, rate-distortion optimization, transform and quantization, and filtering and enhancement. In each part, problem formulation, working flow, recent advances, advantages, and challenges are presented. Fourthly, perceptual coding performances of the latest coding standards and tools are experimentally analyzed. Finally, challenging issues and future opportunities are identified.

KW - Additional Key Words and PhrasesPerceptual video coding

KW - human visual system

KW - just noticeable difference

KW - quality assessment

KW - rate-distortion optimization

KW - visual attention

UR - http://www.scopus.com/inward/record.url?scp=85152604656&partnerID=8YFLogxK

U2 - 10.1145/3571727

DO - 10.1145/3571727

M3 - Journal Article (refereed)

SN - 0360-0300

VL - 55

JO - ACM Computing Surveys

JF - ACM Computing Surveys

IS - 12

M1 - 3571727

ER -

A Survey on Perceptually Optimized Video Coding

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Adaptive Dynamic Range Enhancement Oriented to High Dynamic Display (面向高動態顯示的自適應動態範圍增強)

Cite this

A Survey on Perceptually Optimized Video Coding

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Adaptive Dynamic Range Enhancement Oriented to High Dynamic Display (面向高動態顯示的自適應動態範圍增強)

Cite this