Log-normality and skewness of estimated state/action values in reinforcement learning

Liangpeng ZHANG; Ke TANG; Xin YAO

Log-normality and skewness of estimated state/action values in reinforcement learning

Liangpeng ZHANG, Ke TANG, Xin YAO

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

5 Citations (Scopus)

Abstract

Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path-wise values that follow log-normal distributions. Since log-normal distributions are skewed, the distribution of estimated state/action values can also be skewed, leading to an imbalanced likelihood of under/overestimation. The degree of such imbalance can vary greatly among actions and policies within a single problem instance, making the agent prone to select actions/policies that have inferior expected return and higher likelihood of overestimation. We present a comprehensive analysis to such skewness, examine its factors and impacts through both theoretical and empirical results, and discuss the possible ways to reduce its undesirable effects.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017)
Editors	Ulrike VON LUXBURG, Isabelle GUYON , Samy BENGIO , Hanna WALLACH, Rob FERGUS, S.V.N. VISHWANATHAN, Roman GARNETT
Publisher	Neural Information Processing Systems Foundation
Pages	1805-1815
Number of pages	11
ISBN (Print)	9781510860964
Publication status	Published - Dec 2017
Externally published	Yes
Event	31st Conference on Neural Information Processing Systems - Long Beach, United States Duration: 4 Dec 2017 → 9 Dec 2017

Publication series

Name	Advances in Neural Information Processing Systems
ISSN (Print)	1049-5258

Conference

Conference	31st Conference on Neural Information Processing Systems
Abbreviated title	NIPS 2017
Country/Territory	United States
City	Long Beach
Period	4/12/17 → 9/12/17

Bibliographical note

Publisher Copyright:
© 2017 Neural information processing systems foundation. All rights reserved.

Funding

This paper was supported by Ministry of Science and Technology of China (Grant No. 2017YFB1003102), the National Natural Science Foundation of China (Grant Nos. 61672478 and 61329302), the Science and Technology Innovation Committee Foundation of Shenzhen (Grant No. ZDSYS201703031748284), EPSRC (Grant No. J017515/1), and in part by the Royal Society Newton Advanced Fellowship (Reference No. NA150123).

Access to Document

https://proceedings.neurips.cc/paper_files/paper/2017/file/69a5b5995110b36a9a347898d97a610e-Paper.pdf

Cite this

ZHANG, L., TANG, K., & YAO, X. (2017). Log-normality and skewness of estimated state/action values in reinforcement learning. In U. VON LUXBURG, I. GUYON , S. BENGIO , H. WALLACH, R. FERGUS, S. V. N. VISHWANATHAN, & R. GARNETT (Eds.), Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017) (pp. 1805-1815). (Advances in Neural Information Processing Systems). Neural Information Processing Systems Foundation. https://proceedings.neurips.cc/paper_files/paper/2017/file/69a5b5995110b36a9a347898d97a610e-Paper.pdf

ZHANG, Liangpeng ; TANG, Ke ; YAO, Xin. / Log-normality and skewness of estimated state/action values in reinforcement learning. Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017). editor / Ulrike VON LUXBURG ; Isabelle GUYON ; Samy BENGIO ; Hanna WALLACH ; Rob FERGUS ; S.V.N. VISHWANATHAN ; Roman GARNETT. Neural Information Processing Systems Foundation, 2017. pp. 1805-1815 (Advances in Neural Information Processing Systems).

@inproceedings{2173edf7ca7d44c68c60ae0432f261c1,

title = "Log-normality and skewness of estimated state/action values in reinforcement learning",

abstract = "Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path-wise values that follow log-normal distributions. Since log-normal distributions are skewed, the distribution of estimated state/action values can also be skewed, leading to an imbalanced likelihood of under/overestimation. The degree of such imbalance can vary greatly among actions and policies within a single problem instance, making the agent prone to select actions/policies that have inferior expected return and higher likelihood of overestimation. We present a comprehensive analysis to such skewness, examine its factors and impacts through both theoretical and empirical results, and discuss the possible ways to reduce its undesirable effects.",

author = "Liangpeng ZHANG and Ke TANG and Xin YAO",

note = "Publisher Copyright: {\textcopyright} 2017 Neural information processing systems foundation. All rights reserved.; 31st Conference on Neural Information Processing Systems, NIPS 2017 ; Conference date: 04-12-2017 Through 09-12-2017",

year = "2017",

month = dec,

language = "English",

isbn = "9781510860964",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural Information Processing Systems Foundation",

pages = "1805--1815",

editor = "{VON LUXBURG}, Ulrike and {GUYON }, {Isabelle } and {BENGIO }, {Samy } and WALLACH, {Hanna } and FERGUS, {Rob } and VISHWANATHAN, {S.V.N. } and Roman GARNETT",

booktitle = "Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017)",

}

ZHANG, L, TANG, K & YAO, X 2017, Log-normality and skewness of estimated state/action values in reinforcement learning. in U VON LUXBURG, I GUYON , S BENGIO , H WALLACH, R FERGUS, SVN VISHWANATHAN & R GARNETT (eds), Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017). Advances in Neural Information Processing Systems, Neural Information Processing Systems Foundation, pp. 1805-1815, 31st Conference on Neural Information Processing Systems, Long Beach, California, United States, 4/12/17. <https://proceedings.neurips.cc/paper_files/paper/2017/file/69a5b5995110b36a9a347898d97a610e-Paper.pdf>

Log-normality and skewness of estimated state/action values in reinforcement learning. / ZHANG, Liangpeng; TANG, Ke; YAO, Xin.
Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017). ed. / Ulrike VON LUXBURG; Isabelle GUYON ; Samy BENGIO ; Hanna WALLACH; Rob FERGUS; S.V.N. VISHWANATHAN; Roman GARNETT. Neural Information Processing Systems Foundation, 2017. p. 1805-1815 (Advances in Neural Information Processing Systems).

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

TY - GEN

T1 - Log-normality and skewness of estimated state/action values in reinforcement learning

AU - ZHANG, Liangpeng

AU - TANG, Ke

AU - YAO, Xin

PY - 2017/12

Y1 - 2017/12

N2 - Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path-wise values that follow log-normal distributions. Since log-normal distributions are skewed, the distribution of estimated state/action values can also be skewed, leading to an imbalanced likelihood of under/overestimation. The degree of such imbalance can vary greatly among actions and policies within a single problem instance, making the agent prone to select actions/policies that have inferior expected return and higher likelihood of overestimation. We present a comprehensive analysis to such skewness, examine its factors and impacts through both theoretical and empirical results, and discuss the possible ways to reduce its undesirable effects.

AB - Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path-wise values that follow log-normal distributions. Since log-normal distributions are skewed, the distribution of estimated state/action values can also be skewed, leading to an imbalanced likelihood of under/overestimation. The degree of such imbalance can vary greatly among actions and policies within a single problem instance, making the agent prone to select actions/policies that have inferior expected return and higher likelihood of overestimation. We present a comprehensive analysis to such skewness, examine its factors and impacts through both theoretical and empirical results, and discuss the possible ways to reduce its undesirable effects.

UR - http://www.scopus.com/inward/record.url?scp=85047021495&partnerID=8YFLogxK

M3 - Conference paper (refereed)

SN - 9781510860964

T3 - Advances in Neural Information Processing Systems

SP - 1805

EP - 1815

BT - Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017)

A2 - VON LUXBURG, Ulrike

A2 - GUYON , Isabelle

A2 - BENGIO , Samy

A2 - WALLACH, Hanna

A2 - FERGUS, Rob

A2 - VISHWANATHAN, S.V.N.

A2 - GARNETT, Roman

PB - Neural Information Processing Systems Foundation

T2 - 31st Conference on Neural Information Processing Systems

Y2 - 4 December 2017 through 9 December 2017

ER -

ZHANG L, TANG K, YAO X. Log-normality and skewness of estimated state/action values in reinforcement learning. In VON LUXBURG U, GUYON I, BENGIO S, WALLACH H, FERGUS R, VISHWANATHAN SVN, GARNETT R, editors, Advances in Neural Information Processing Systems, 30 : 31st Annual Conference on Neural Information Processing Systems (NIPS 2017). Neural Information Processing Systems Foundation. 2017. p. 1805-1815. (Advances in Neural Information Processing Systems).

Log-normality and skewness of estimated state/action values in reinforcement learning

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this