An efficient mode decision algorithm for H.264/AVC encoding optimization

Hanli WANG, Sam KWONG, Chi-Wah KOK

Research output: Journal PublicationsJournal Article (refereed)peer-review

89 Citations (Scopus)


The H.264 video coding standard significantly outperforms previous standards in terms of coding efficiency. However, this comes as a cost of extremely high computational complexity due to mode decision where variable block size Motion Estimation (ME) is employed. In this paper, we propose an efficient algorithm to jointly optimize mode decision and ME. A theoretical analysis is performed to study the sufficient condition to detect all-zero blocks in H.264, and thus adaptive thresholds are derived to early terminate mode decision and ME. Besides the aforementioned early termination technique, the proposed algorithm also introduces temporal-spatial checking, thresholds based prediction and monotonic error surface based prediction methods to skip checking unnecessary modes. Experimental results demonstrate that the proposed algorithm can significantly reduce the computational complexity of H.264 encoding while maintaining almost the same Rate Distortion (RD) performance as the original encoder. © 2007 IEEE.
Original languageEnglish
Pages (from-to)882-888
JournalIEEE Transactions on Multimedia
Issue number4
Publication statusPublished - Jun 2007
Externally publishedYes

Bibliographical note

This work was supported by City University of Hong Kong Strategic Grant 7001955.


  • Early termination
  • H.264
  • Integer DCT
  • Mode decision
  • Motion estimation


Dive into the research topics of 'An efficient mode decision algorithm for H.264/AVC encoding optimization'. Together they form a unique fingerprint.

Cite this