Improving Generalization in Reinforcement Learning-Based Trading by Using a Generative Adversarial Market Model

Chia-Hsuan Kuo, Chiao-Ting Chen, Sin-Jing Lin, Szu-Hao Huang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

21 Scopus citations

Abstract

With the increasing sophistication of artificial intelligence, reinforcement learning (RL) has been widely applied to portfolio management. However, shortcomings remain. Specifically, because the training environment of an RL-based portfolio optimization framework is usually constructed based on historical price data in the literature, the agent potentially 1) violates the definition of a Markov decision process (MDP), 2) ignores their own market impact, or 3) fails to account for causal relationships within interaction processes; these ultimately lead the agent to make poor generalizations. To surmount these problems-specifically, to help the RL-based portfolio agent make better generalizations-we introduce an interactive training environment that leverages a generative model, called the limit order book-generative adversarial model (LOB-GAN), to simulate a financial market. Specifically, the LOB-GAN models market ordering behavior, and LOB-GAN's generator is utilized as a market behavior simulator. A simulated financial market, called Virtual Market, is constructed by the market behavior simulator in conjunction with a realistic security matching system. Virtual Market is then leveraged as an interactive training environment for the RL-based portfolio agent. The experimental results demonstrate that our framework improves out-of-sample portfolio performance by 4%, which is superior to other generalization strategies.

Original languageEnglish
Pages (from-to)50738-50754
Number of pages17
JournalIEEE Access
Volume9
DOIs
StatePublished - Mar 2021

Keywords

  • Portfolios
  • Training
  • Optimization
  • Topology
  • Data models
  • Stock markets
  • Network topology
  • Artificial market simulation
  • portfolio management
  • reinforcement learning

Fingerprint

Dive into the research topics of 'Improving Generalization in Reinforcement Learning-Based Trading by Using a Generative Adversarial Market Model'. Together they form a unique fingerprint.

Cite this