Improving Generalization in Reinforcement Learning-Based Trading by Using a Generative Adversarial Market Model

Chia-Hsuan Kuo, Chiao-Ting Chen, Sin-Jing Lin, Szu-Hao Huang*

*Corresponding author for this work

    Research output: Contribution to journalArticlepeer-review

    2 Scopus citations

    Abstract

    With the increasing sophistication of artificial intelligence, reinforcement learning (RL) has been widely applied to portfolio management. However, shortcomings remain. Specifically, because the training environment of an RL-based portfolio optimization framework is usually constructed based on historical price data in the literature, the agent potentially 1) violates the definition of a Markov decision process (MDP), 2) ignores their own market impact, or 3) fails to account for causal relationships within interaction processes; these ultimately lead the agent to make poor generalizations. To surmount these problems-specifically, to help the RL-based portfolio agent make better generalizations-we introduce an interactive training environment that leverages a generative model, called the limit order book-generative adversarial model (LOB-GAN), to simulate a financial market. Specifically, the LOB-GAN models market ordering behavior, and LOB-GAN's generator is utilized as a market behavior simulator. A simulated financial market, called Virtual Market, is constructed by the market behavior simulator in conjunction with a realistic security matching system. Virtual Market is then leveraged as an interactive training environment for the RL-based portfolio agent. The experimental results demonstrate that our framework improves out-of-sample portfolio performance by 4%, which is superior to other generalization strategies.

    Original languageEnglish
    Pages (from-to)50738-50754
    Number of pages17
    JournalIEEE Access
    Volume9
    DOIs
    StatePublished - Mar 2021

    Keywords

    • Portfolios
    • Training
    • Optimization
    • Topology
    • Data models
    • Stock markets
    • Network topology
    • Artificial market simulation
    • portfolio management
    • reinforcement learning

    Fingerprint

    Dive into the research topics of 'Improving Generalization in Reinforcement Learning-Based Trading by Using a Generative Adversarial Market Model'. Together they form a unique fingerprint.

    Cite this