Sang, J., Khan, Zaki Ahmed, Yin, H. and Wang, Y. (2023) Reward shaping using directed graph convolution neural networks for reinforcement learning and games. Frontiers in Physics: Section of Social Physics, 11. pp. 1-9. ISSN 2296-424X
|
Text
Reward shaping using directed graph convolution neural networks for reinforcement learning and games.pdf - Published Version Available under License Creative Commons Attribution. Download (3MB) | Preview |
Abstract
Game theory can employ reinforcement learning algorithms to identify the optimal policy or equilibrium solution. Potential-based reward shaping (PBRS) methods are prevalently used for accelerating reinforcement learning, ensuring the optimal policy remains consistent. Existing PBRS research performs message passing based on graph convolution neural networks (GCNs) to propagate information from rewarding states. However, in an irreversible time-series reinforcement learning problem, undirected graphs will not only mislead message-passing schemes but also lose a distinctive direction structure. In this paper, a novel approach called directed graph convolution neural networks for reward shaping φDCN has been proposed to tackle this problem. The key innovation of φDCN is the extension of spectral-based undirected graph convolution to directed graphs. Messages can be efficiently propagated by leveraging a directed graph Laplacian as a substitute for the state transition matrix. As a consequence, potential-based reward shaping can then be implemented by the propagated messages. The incorporation of temporal dependencies between states makes φDCN more suitable for real-world scenarios than existing potential-based reward shaping methods based on undirected graph convolutional networks. Preliminary experiments demonstrate that the proposed φDCN exhibits a substantial improvement compared to other competing algorithms on both Atari and MuJoCo benchmarks.
Item Type: | Article |
---|---|
Additional Information: | © 2023 Sang, Ahmad Khan, Yin and Wang. |
Uncontrolled Discrete Keywords: | Markov decision process, reinforcement learning, directed graph convolutional network, reward shaping, game |
Divisions: | College of Business, Psychology and Sport > Worcester Business School |
Related URLs: | |
Copyright Info: | Open Access Article |
Depositing User: | Katherine Small |
Date Deposited: | 22 Jan 2024 10:20 |
Last Modified: | 22 Jan 2024 10:59 |
URI: | https://eprints.worc.ac.uk/id/eprint/13513 |
Actions (login required)
![]() |
View Item |