Reinforcement Learning in Finance: Resources and Expert Advice from Paul Bilokon

14 min read

Reinforcement learning (RL) is one of the most exciting areas of Machine Learning, especially when applied to trading. RL is so appealing because it allows you to optimise strategies and enhance decision-making in ways that traditional methods can’t.

One of its biggest advantages?

You don’t have to spend a lot of time manually training the model. Instead, RL learns and makes trading decisions on its own (depending on feedback once received), continuously adjusting as per the dynamism of the market. This efficiency and autonomy are why RL is becoming so popular in finance.

As per the news, “The global Reinforcement Learning market was valued at $2.8 billion in 2022 and is projected to reach $88.7 billion by 2032, growing at a CAGR of 41.5% from 2023 to 2032.⁽¹⁾

Please note that we have prepared the content in this article almost entirely from Dr Paul Bilokon’s QuantInsti webinar. You can watch the webinar (below) if you wish to.

About the Speaker

Dr. Paul Bilokon, CEO and Founder of Thalesians Ltd, is a prominent figure in quantitative finance, algorithmic trading, and machine learning. He leads innovation in financial technology through his role at Thalesians Ltd and serves as the Chief Scientific Advisor at Thalesians Marine Ltd. In addition to his industry work, he heads the faculty at the Machine Learning Institute and the Quantitative Developer Certificate, playing a key role in shaping the future of quantitative finance education.

In this blog, we will first explore key research papers that will help you learn Reinforcement Learning in finance along with the latest advancements in RL applied to finance.

We will then navigate through some good books in the field.

Finally, we will take a look at valuable insights covered in the FAQ session with Paul Bilokon, where he answers an assortment of questions about reinforcement learning and its impact on trading strategies.

Let’s get started on this learning journey as this blog covers the following for learning Reinforcement Learning in Finance in depth:


Key Research Papers

Below are the key research papers recommended by Paul on Reinforcement Learning in finance.

Apart from the above-mentioned research papers which Paul recommends, let us also look at some other research papers below that are quite beneficial for learning Reinforcement Learning in finance.

**Note: The research papers below are not from the webinar video featuring Paul Bilokon.**

  • Deep Reinforcement Learning for Algorithmic Trading (Link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3812473) by Álvaro Cartea, Sebastian Jaimungal and Leandro Sánchez-Betancourt explains how reinforcement learning techniques like double deep Q networks (DDQN) and reinforced deep Markov models (RDMMs) are used to create optimal statistical arbitrage strategies in foreign exchange (FX) triplets. The paper also demonstrates their effectiveness through simulations of exchange rate models.
  • Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy (Link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3690996) by Hongyang Yang, Xiao-Yang Liu, Shan Zhong and Anwar Walid covers the explanation of an ensemble stock trading strategy that uses deep reinforcement learning to maximise investment returns. By combining three actor-critic algorithms (PPO, A2C, and DDPG), it creates a robust trading strategy that outperforms individual algorithms and traditional baselines in risk-adjusted returns, tested on Dow Jones stocks.
  • Reinforcement Learning Pair Trading: A Dynamic Scaling Approach (Link: https://arxiv.org/pdf/2407.16103) by Hongshen Yang and Avinash Malik explores the use of reinforcement learning (RL) combined with pair trading to enhance cryptocurrency trading. By testing RL techniques on BTC-GBP and BTC-EUR pairs, it demonstrates that RL-based strategies significantly outperform traditional pair trading methods, yielding annualised profits between 9.94% and 31.53%.
  • Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (Link: https://ar5iv.labs.arxiv.org/html/2111.09395) by Xiao-Yang Liu, Hongyang Yang, Christina Dan Wang and Jiechao Gao introduces FinRL, the first open-source framework designed to help quantitative traders apply deep reinforcement learning (DRL) to trading strategies, overcoming the challenges of error-prone programming and debugging. FinRL offers a full pipeline with modular, customisable algorithms, simulations of various markets, and hands-on tutorials for tasks like stock trading, portfolio allocation, and cryptocurrency trading.
  • Deep Reinforcement Learning Approach for Trading Automation in The Stock Market (Link: https://arxiv.org/abs/2208.07165) by Taylan Kabbani and Ekrem Duman covers how Deep Reinforcement Learning (DRL) algorithms can automate profit generation in the stock market by combining price prediction and portfolio allocation into a unified process. It formulates the trading problem as a Partially Observed Markov Decision Process (POMDP) and demonstrates the effectiveness of the TD3 algorithm, achieving a 2.68 Sharpe Ratio, while highlighting DRL’s superiority over traditional machine learning approaches in financial markets.

Now let us find out about all those books that Paul recommends for learning Reinforcement Learning in finance.


Useful Books

You can see the list of books below:

  • Reinforcement Learning: An Introduction by Sutton and Barto is a foundational book on reinforcement learning, covering essential concepts that can be applied to various domains, including finance.
RL An introduction
  • Algorithms for Reinforcement Learning by Csaba Szepesvári offers a deeper dive into the algorithms driving RL, helpful for those interested in the technical side of financial applications.
Algorithms for RL
  • Reinforcement Learning and Optimal Control by Dimitri Bertsekas explores Reinforcement Learning, approximate dynamic programming, and other methods to bridge optimal control and Artificial Intelligence, with a focus on approximation techniques across various types of problems and solution methods.
RL and optimal control
  • Reinforcement Learning Theory by Agarwal, Jiang, and Sun is a newer work offering advanced insights into RL theory.

https://rltheorybook.github.io/rltheorybook_AJKS.pdf

  • Deep Reinforcement Learning Hands-On by Maxim Lapan how to use deep learning (DL) and Deep Reinforcement Learning (RL) to solve complex problems, covering key methods and applications, including training agents for Atari games, stock trading, and AI-driven chatbots. Ideal for those familiar with Python and basic DL concepts, it offers practical insights into the latest algorithms and industry developments.
Deep RL Hands-on
  • Deep Reinforcement Learning in Action by Alexander Zai and Brandon Brown explains how to develop AI agents that learn from feedback and adapt to their environments, using techniques like deep Q-networks and policy gradients, supported by practical examples and Jupyter Notebooks. Suitable for readers with intermediate Python and deep learning skills, the book includes access to a free eBook.
Deep RL in action
  • Machine Learning in Finance by Matthew Dixon, Igor Halperin and Paul Bilokon offers a comprehensive guide to applying Machine Learning in finance, combining theories from econometrics and stochastic control to help readers select optimal algorithms for financial modelling and decision-making. Targeted at advanced students and professionals, it covers supervised learning for cross-sectional and time series data, as well as reinforcement learning in finance, with practical Python examples and exercises.
ML in Finance
  • Machine Learning and Big Data with Kdb+ by Bilokon, Novotny, Galiotos, and Deleze, focuses on handling vast datasets for finance, which is essential for those working with real-time market data.
ML and big data
  • Essential concepts like Multi-Armed Bandits, Markov decision processes, and dynamic programming form the basis for many RL strategies in finance. These concepts enable the exploration of decision-making under uncertainty, a core element in financial modelling.

Books on Multi-Armed Bandits

Books on Markov decision processes and dynamic programming

These resources provide a solid foundation for understanding and applying Reinforcement Learning in finance, offering theoretical insights as well as practical applications for real-world challenges like hedging, wealth management, and optimal execution.

Let us check out some blogs next that are quite informative as they cover essential topics on Reinforcement Learning in finance.


Blogs

Below are some of the blogs you can read.

This blog consists of information on how Reinforcement Learning can be applied to finance, and why it might be one of the most transformative technologies in this space. The blog is based on the podcast by Dr. Yves J. Hilpisch which he covered in his podcast. Dr. Yves J. Hilpisch is a renowned figure in the world of quantitative finance, known for championing the use of Python in financial trading and algorithmic strategies.

This blog post covers how Multiagent Reinforcement Learning can be used to develop optimal trading strategies by simulating competitive agents. It demonstrates the effectiveness of competing agents in outperforming noncompeting agents when trading in a simulated stock environment.

This blog covers the development of a Reinforcement Learning system that provides dynamic investment recommendations to maximise returns in a stock portfolio. It explains how the system handles complex market conditions, manages risk, and uses approximation methods to optimise decision-making in scarce environments.

Lastly, you can see the questions that the webinar audience asked Paul.


FAQs with Paul Bilokon: Expert Insights

Below are a few interesting questions the audience asked and very useful answers by Paul.

Q: How can Reinforcement Learning be useful in trading with low signal-to-noise ratios?

A: Yes, reinforcement learning can indeed be useful in finance. However, it's important to consider that finance often has a very low signal-to-noise ratio and non-stationarity, meaning the statistical properties of financial data change over time. These conditions aren't unique to finance, as they also appear in fields like life sciences and physical sciences with high stochasticity. I’ve written several papers addressing how to handle non-stationarity and low signal-to-noise ratio environments; they can be found on my SSRN page.

If you type “Paul Bilokon papers” on Google, you will see a list of SSRN research papers. The ones published in 2024 have a lot of such papers that explain how to deal with non-stationarity in the presence of low signal to noise ratio.

Q: Can Supervised Learning models like Black-Scholes guide Reinforcement Learning in trading?

A: Yes, they can. For instance, you can use the Black-Scholes model or a classical PDE solver to train reinforcement learning agents initially. Afterwards, you can improve your model by using real data to fine-tune the training. This approach combines insights from classical models with the flexibility of reinforcement learning.

Q: How important is coding experience for machine learning and reinforcement learning in finance?

A: Practical experience in programming is crucial. Those working in reinforcement learning or machine learning, in general, should be able to code quickly and efficiently. Many experts in reinforcement learning, like David Silver, come from software development backgrounds, often with experience in video game development. Building proficiency in programming can significantly enhance one's ability to handle data and develop sophisticated ML solutions.

Q: Is market and signal selection in a financial model a feature selection problem?

A: Yes, it can be viewed as a feature selection problem. You face the classic bias-variance trade-off. Using all features can introduce noise, while reducing features can help manage variance, but might increase bias. An effective feature selection algorithm will help maintain a balance, reducing variance without introducing too much bias and thus improving mean squared error.

Q: What are the top three trading strategies for quant researchers to explore?

A: Basic trading strategies from textbooks, such as momentum and mean reversion, may not work directly in practice, as many have been arbitraged away due to widespread use. Instead, understanding the statistical and market principles behind these strategies can inspire more sophisticated methods. Techniques like deep learning, if properly controlled for complexity and overfitting, could also help in feature selection and decision-making.

Q: Can options trading strategies achieve high AUM like mutual funds?

A: Options trading and mutual funds represent different financial activities and they are not directly comparable. For instance, selling options can expose one to high risk, so it’s often reserved for professionals due to the potential for unlimited downside. While options trading can yield higher fees, it’s essential to understand its inherent risks, such as the volatility risk premium.

Q: What happens when multiple traders use the same reinforcement learning strategy in the market?

A: If the market has high capacity and both are trading small sizes, they may not impact each other significantly. However, if the strategy’s capacity is low, competing participants can cause alpha decay, reducing profitability. Generally, once a strategy becomes well-known, overuse can lead to diminished returns.

Q: Is there a “Hugging Face” equivalent for reinforcement learning with pre-trained models?

A: OpenAI Gym provides a variety of classical environments for reinforcement learning and offers standard models like Deep Q-Learning and Expected SARSA. OpenAI Gym allows users to apply and refine models on these environments and then extend them to more complex real-world applications.

Q: How can Machine Learning enhance fundamental analysis for value investing?

A: Large Language Models (LLMs) can now process extensive unstructured data, such as text. Using a framework like LangChain with an LLM enables the automated processing of financial documents, like PDFs, to analyse fundamentals. Combining this with ML models can help identify undervalued, high-quality stocks based on fundamental analysis.

Courses by QuantInsti

**Note: This topic is not addressed in the webinar video featuring Paul Bilokon.**

Additionally, the following courses by QuantInsti cover Reinforcement Learning in finance.

This free course introduces you to the application of machine learning in trading, focusing on the implementation of various algorithms using financial market data. You will explore different research studies and gain a comprehensive understanding of this specialised area.

Utilise reinforcement learning to develop, backtest, and execute a trading strategy with two deep-learning neural networks and replay memory. This hands-on Python course emphasises quantitative analysis of returns and risks, culminating in a capstone project focused on financial markets.

If you are interested in using AI to determine optimal investments in Gold or Microsoft stocks, this course is the one for you. This course leverages LSTM networks to teach fundamental portfolio management, including mean-variance optimisation, AI algorithm applications, walk-forward optimisation, hyperparameter tuning, and real-world portfolio management. Also, you will get hands-on experience through live trading templates and capstone projects.


Conclusion

This blog explored key resources, including research papers, books, and expert insights from Paul Bilokon, to help you dive deeper into the world of RL in finance. Whether you are looking to optimise trading strategies or explore cutting-edge AI-driven solutions, the resources discussed provide a comprehensive foundation. As you continue your learning journey, leveraging these resources will equip you with the necessary tools to excel in the field of quantitative finance and algorithmic trading using reinforcement learning.

You can learn Reinforcement Learning in depth with the course on Deep Reinforcement Learning in Trading. With this course, you can take your trading skills to the next level as you will learn to apply reinforcement learning to create, backtest, and trade strategies. Further, you will learn to master quantitative analysis of returns and risks, ending the course with implementable techniques and a capstone project in financial markets.


File in the download:

  • PPT By Paul Bilokon


Compiled by: Chainika Thakar


Disclaimer: All data and information provided in this article are for informational purposes only. QuantInsti® makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information in this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis..

Free Algo Trading Beginner Courses