Reinforcement Learning for Finance: Solve Problems in Finance with CNN and RNN Using the TensorFlow Library

435 Pages·2023·17.615 MB·English

by  Samit Ahlawat

Reinforcement
Learning for
Finance
Solve Problems in Finance with
CNN and RNN Using the
TensorFlow Library
—
Samit Ahlawat
Reinforcement
Learning for Finance
Solve Problems in Finance
with CNN and RNN Using
the TensorFlow Library
Samit Ahlawat
Reinforcement Learning for Finance: Solve Problems in Finance with CNN
and RNN Using the TensorFlow Library
Samit Ahlawat
Irvington, NJ, USA
ISBN-13 (pbk): 978-1-4842-8834-4 ISBN-13 (electronic): 978-1-4842-8835-1
https://doi.org/10.1007/978-1-4842-8835-1
Copyright © 2023 by Samit Ahlawat
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or
part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way,
and transmission or information storage and retrieval, electronic adaptation, computer software,
or by similar or dissimilar methodology now known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark
symbol with every occurrence of a trademarked name, logo, or image we use the names, logos,
and images only in an editorial fashion and to the benefit of the trademark owner, with no
intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if
they are not identified as such, is not to be taken as an expression of opinion as to whether or not
they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of
publication, neither the authors nor the editors nor the publisher can accept any legal
responsibility for any errors or omissions that may be made. The publisher makes no warranty,
express or implied, with respect to the material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Celestin Suresh John
Development Editor: Laura Berendson
Coordinating Editor: Mark Powers
Cover designed by eStudioCalamar
Cover image by Joel Filipe on Unsplash (www.unsplash.com)
Distributed to the book trade worldwide by Apress Media, LLC, 1 New York Plaza, New York, NY
10004, U.S.A. Phone 1-800-SPRINGER, fax [phone], e-mail [email protected],
or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member
(owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance
Inc is a Delaware corporation.
For information on translations, please e-mail [email protected]; for
reprint, paperback, or audio rights, please e-mail [email protected].
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook
versions and licenses are also available for most titles. For more information, reference our Print
and eBook Bulk Sales web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is
available to readers on GitHub (https://github.com/Apress). For more detailed information,
please visit http://www.apress.com/source-code.
Printed on acid-free paper
To my family and friends without whose support this book
would not have been possible.
Table of Contents
About the Author ���������������������������������������������������������������������������������ix
Acknowledgments �������������������������������������������������������������������������������xi
Preface ����������������������������������������������������������������������������������������������xiii
Introduction ����������������������������������������������������������������������������������������xv
Chapter 1: Overview �����������������������������������������������������������������������������1
1.1 Methods for Training Neural Networks ...........................................................2
1.2 Machine Learning in Finance ..........................................................................3
1.3 Structure of the Book ......................................................................................4
Chapter 2: Introduction to TensorFlow �������������������������������������������������5
2.1 Tensors and Variables .....................................................................................5
2.2 Graphs, Operations, and Functions ...............................................................11
2.3 Modules ........................................................................................................14
2.4 Layers ...........................................................................................................17
2.5 Models ..........................................................................................................25
2.6 Activation Functions ......................................................................................33
2.7 Loss Functions ..............................................................................................37
2.8 Metrics ..........................................................................................................46
2.9 Optimizers .....................................................................................................77
2.10 Regularizers ................................................................................................96
2.11 TensorBoard ..............................................................................................120
v
Table of ConTenTs
2.12 Dataset Manipulation ................................................................................122
2.13 Gradient Tape ............................................................................................126
Chapter 3: Convolutional Neural Networks ���������������������������������������139
3.1 A Simple CNN ..............................................................................................140
3.2 Neural Network Layers Used in CNNs .........................................................148
3.3 Output Shapes and Trainable Parameters of CNNs .....................................150
3.4 Classifying Fashion MNIST Images .............................................................152
3.5 Identifying Technical Patterns in Security Prices ........................................159
3.6 Using CNNs for Recognizing Handwritten Digits .........................................172
Chapter 4: Recurrent Neural Networks ���������������������������������������������177
4.1 Simple RNN Layer .......................................................................................178
4.2 LSTM Layer .................................................................................................182
4.3 GRU Layer ...................................................................................................186
4.4 Customized RNN Layers ..............................................................................188
4.5 Stock Price Prediction .................................................................................190
4.6 Correlation in Asset Returns .......................................................................207
Chapter 5: Reinforcement Learning Theory ��������������������������������������233
5.1 Basics .........................................................................................................234
5.2 Methods for Estimating the Markov Decision Problem ...............................240
5.3 Value Estimation Methods ...........................................................................241
5.3.1 Dynamic Programming .......................................................................242
5.3.2 Generalized Policy Iteration ................................................................265
5.3.3 Monte Carlo Method ...........................................................................277
5.3.4 Temporal Difference (TD) Learning .....................................................284
5.3.5 Cartpole Balancing .............................................................................305
vi
Table of ConTenTs
5.4 Policy Learning............................................................................................319
5.4.1 Policy Gradient Theorem .....................................................................319
5.4.2 REINFORCE Algorithm .........................................................................321
5.4.3 Policy Gradient with State-Action Value Function Approximation ......323
5.4.4 Policy Learning Using Cross Entropy ..................................................325
5.5 Actor-Critic Algorithms ................................................................................326
5.5.1 Stochastic Gradient–Based Actor-Critic Algorithms ...........................329
5.5.2 Building a Trading Strategy ................................................................330
5.5.3 Natural Actor-Critic Algorithms ...........................................................346
5.5.4 Cross Entropy–Based Actor-Critic Algorithms ....................................347
Chapter 6: Recent RL Algorithms������������������������������������������������������349
6.1 Double Deep Q-Network: DDQN ..................................................................349
6.2 Balancing a Cartpole Using DDQN ...............................................................353
6.3 Dueling Double Deep Q-Network ................................................................356
6.4 Noisy Networks ...........................................................................................357
6.5 Deterministic Policy Gradient ......................................................................359
6.5.1 Off-Policy Actor-Critic Algorithm .........................................................360
6.5.2 Deterministic Policy Gradient Theorem ..............................................361
6.6 Trust Region Policy Optimization: TRPO ......................................................362
6.7 Natural Actor-Critic Algorithm: NAC .............................................................368
6.8 Proximal Policy Optimization: PPO ..............................................................369
6.9 Deep Deterministic Policy Gradient: DDPG ..................................................370
6.10 D4PG .........................................................................................................373
6.11 TD3PG........................................................................................................376
6.12 Soft Actor-Critic: SAC ................................................................................379
vii
Table of ConTenTs
6.13 Variational Autoencoder ............................................................................384
6.14 VAE for Dimensionality Reduction .............................................................389
6.15 Generative Adversarial Networks ..............................................................399
Bibliography �������������������������������������������������������������������������������������403
Index �������������������������������������������������������������������������������������������������411
viii
About the Author
Samit Ahlawat is Senior Vice President in
Quantitative Research, Capital Modeling, at
JPMorgan Chase in New York, USA. In his
current role, he is responsible for building
trading strategies for asset management
and for building risk management models.
His research interests include artificial
intelligence, risk management, and
algorithmic trading strategies. He has given CQF Institute talks on artificial
intelligence, has authored several research papers in finance, and holds a
patent for facial recognition technology. In his spare time, he contributes
to open source code.
ix
Acknowledgments
I would like to express my heartfelt appreciation for my friends and
coworkers, in academia and the workplace, who encouraged me to write
this book.
xi