Contextual reinforcement learning for market making via multi-task self-supervised learning

Market makers play important roles in modern financial transactions by simultaneously providing buy and sell limit orders to offer liquidity and earn the bid–ask spread. In recent years, several deep reinforcement learning techniques have been developed for high-frequency quantitative trading tasks such as market making. However, due to the uncertainty and variability of financial markets, reinforcement learning for trading needs to overcome two major challenges: concept drift and insufficient generalization ability. In this study, we propose a novel Contextual Reinforcement Learning framework for training a generalized market-making trading strategy, which utilizes financial market sentiment indicators as contextual information, enabling reinforcement learning agents to adjust their order placement decisions according to different market contexts. Subsequently, we introduce a Self-Supervised Market Context Network that incorporates multiple auxiliary tasks with financial domain knowledge, which can further enhance the agent’s sensitivity and understanding of the market environment. Furthermore, we adopt the replay-based method from Continual Learning, performing policy fine-tuning by selecting appropriate historical environments, effectively addressing issues of concept drift and catastrophic forgetting. We conducted experiments on real-world tick-level data from the Taiwan Stock Exchange Capitalization Weighted Stock Index (TAIEX) options. Under the same inventory risk, our strategy consistently outperformed state-of-the-art baselines in terms of several evaluation metrics, demonstrating superior profitability per unit of risk. This demonstrates that our proposed framework enables the model to adapt to out-of-sample markets and can also avoid risk and achieve steady profits even in extreme market environments.