Automated stock trading and Portfolio optimization using XCS

100 Pages·2008·1.24 MB·English

Automated Stock Trading and Portfolio
Optimization Using XCS Trader and Technical
Analysis
Anil Chauhan
[email protected]
Master of Science
Artificial Intelligence
School of Informatics
University of Edinburgh
2008
Abstract
Financial market is highly dynamic system for which finding underlying price pattern is highly
complex. We have extended the previous work done on automatic stock trading using extended
classifier system (XCS) by implementing Q (1) and Q (λ) Reinforcement Learning algorithm.
We developed 14 XCS agents using different technical indicators like Moving
averages,RSI,CMF,SAR,ADX etc. We showed that by modeling financial prediction as single
step reinforcement learning problem and using the concept of delayed reward for checking
correctness of action taken, all the benchmarks strategies like buy and hold, 'keeping money in
bank' etc could be beaten. We have also shown that stock price movement is co-related with
other day price movement and reformulated the financial forecasting as a multi step process.
We introduced the concept of passive set and found that multi step problem formulation gives
best results. Q learning gave 18% better performance than single step reward only RL. Finally
we build a portfolio management and optimization system which learns online and does
monthly or quarterly rebalancing using the best trader to trade. The results showed that
reacting to the market dynamics doesn’t necessarily give us the best result. We showed that
such a system give us average performance between the best trader and the worst trader. We
also employed different trading strategies like “using more than 1 best agent” and “mean
reversal strategy” to do portfolio optimization.
ii
Acknowledgements
I would like to thank my supervisor Sonia Schulenburg for introducing me to the world of
Finance and Classifier Systems and for giving constant feed back on my project. I would also
like to thank Abu ul Hassan for sharing previous version of XCS java code with me. Many
thanks to my friend Santosh for reviewing my initial draft of thesis and sharing ideas on the
same.
iii
Declaration
I declare that this thesis was composed by me, that the work contained herein is my own except
where explicitly stated otherwise in the text, and that thesis work has not been submitted for
any other degree or professional qualification except as specified.
(Anil Chauhan
[email protected])
iv
Table of Contents
1 Introduction..............................................................................................................................1
1.1 Introduction and Purpose....................................................................................................1
1.2 Motivation...........................................................................................................................2
1.3 Objective ............................................................................................................................3
1.4 Outline................................................................................................................................4
2 Background & Related Work.................................................................................................5
2.1 Background.........................................................................................................................5
2.1.1 Market Efficiency........................................................................................................5
2.1.1.1 Version of Efficient Market Hypothesis (EMH)..................................................5
2.1.2 Technical Analysis.......................................................................................................6
2.1.3 The Portfolio:..............................................................................................................6
2.1.3.1 Why do we need Portfolio?..................................................................................7
2.1.3.2 Portfolio Management:.........................................................................................7
2.2 Related Work......................................................................................................................7
2.2.1 Machine Learning in Finance and Portfolio Management.........................................8
2.3 XCS Introduction from Stock Trading Perspective..........................................................10
2.3.1 XCS Input and Output...............................................................................................11
2.3.2 XCS Frame Work [15]..............................................................................................12
2.3.3 XCS Learning Cycle..................................................................................................13
2.3.3.1 Updating XCS Parameters..................................................................................15
2.3.3.2 Genetic Algorithm role and rule evolution[15]..................................................15
2.3.4 Deviation from other LCS based Systems.................................................................16
2.3.5 Mind of XCS System..................................................................................................17
3 Implementation .....................................................................................................................18
3.1 Technical Analysis Usage in XCS....................................................................................18
3.1.1 Description of individual technical Indicators..........................................................19
3.1.2 Combining different technical indicators and working mechanism of different
Agents.................................................................................................................................24
v
3.1.2.1 Composition of 14 Agents:.................................................................................25
3.1.3 Advantage and “Scope of Improvement” of current approach................................26
3.2 Improving the learning of eXtended Classifier System....................................................27
3.2.1 Classifiers in multi step Reinforcement learning problems......................................28
3.2.2 Implementing Q learning in Classifier......................................................................30
3.2.3 Eligibility trace and Watkins's Q(λ) .........................................................................30
4 Experimentation.....................................................................................................................32
4.1 FTSE data and Stability of the XCS System....................................................................32
4.1.1 FTSE Data.................................................................................................................32
4.1.2 Stability of the XCS System.......................................................................................32
4.2 Comparative Study of the 3 different Algorithm..............................................................34
4.2.1 Setting the parameters for the experiments...............................................................34
4.2.1.1 Setting Initial Exploration Rate..........................................................................35
4.2.1.2 Setting discount rate (gamma)............................................................................36
4.2.1.3 Setting Trace Decay Parameter...........................................................................37
4.3 Experimental Results for 3 learning Algorithm................................................................37
4.3.1 Observations:............................................................................................................39
4.4 Fault in the previous Reward giving strategy...................................................................40
4.4.1 Experiments with improved delayed reward Strategy...............................................40
4.4.1.1 Setting Initial Exploration Rate..........................................................................41
4.4.1.2 Setting Discount Rate (gamma)..........................................................................42
4.4.1.3 Setting Trace Decay (λ)......................................................................................42
4.4.2 Results with new delayed reward strategy................................................................43
4.4.2.1 Observations:......................................................................................................45
4.4.3 Experimental Results for all 3 learning algorithm with new delayed reward strategy
............................................................................................................................................48
4.4.3.1 Observations.......................................................................................................51
4.4.4 Fault in Q (1) learning ............................................................................................51
4.4.5 Experimentation with passive Set..............................................................................52
4.4.5.1 Finding optimum parameters..............................................................................52
4.4.5.2 Observations for Passive set...............................................................................55
vi
5 Implementation & Experimentation-Portfolio Optimization............................................58
5.1 Implementation.................................................................................................................59
5.2 Portfolio Performance:......................................................................................................60
5.3 Results...............................................................................................................................61
Portfolio Management Results –.......................................................................................62
5.3.1.1 Observations: Portfolio Management results......................................................63
5.3.1.2 Analysis and Comments on Portfolio Management System..............................63
5.3.2 Change of Portfolio construction Strategy................................................................64
5.3.2.1 Results of Portfolio Management using more than 1 Agent...............................65
5.4 Experimentation with Portfolio Management taking few best companies in the Portfolio
................................................................................................................................................65
5.4.1 Steps Followed..........................................................................................................66
5.4.2 Results.......................................................................................................................66
5.4.3 Observation:..............................................................................................................66
5.4.4 Experimentation with Mean Reversal Strategy.........................................................67
5.4.4.1 Results.................................................................................................................67
5.4.4.2 Observations.......................................................................................................68
6 Conclusion & Future Work..................................................................................................69
6.1 Conclusion .......................................................................................................................69
6.2 Future Work......................................................................................................................71
Bibliography..............................................................................................................................72
Appendix....................................................................................................................................76
vii
List of Figures
Figure1 XCS Frame Work ......................................................................................................13
Figure2: Voting Strategy .........................................................................................................17
Figure3: Q-learning: An off-policy TD control algorithm. ..................................................29
Figure4:Back ward view of eligibility trace...........................................................................30
Figure5: Tabular version of Watkins's Q (λ) algorithm.......................................................31
Figure6: Mean Performance of 5 companies for different number of runs........................34
Figure7: DSGI Price Chart .....................................................................................................46
Figure8: DSGI, Wealth Chart of agents with old reward strategy......................................46
Figure9: DSGI, Agent’s performance with new delayed reward giving strategy...............47
Figure10: LAND, Price Chart.................................................................................................47
Figure11: LAND, Meta Agents wealth chart with old reward strategy..............................48
Figure12: LAND, Meta Agents wealth chart with new delayed reward strategy...............48
Figure13: Portfolio Management Results...............................................................................62
Figure14: Portfolio management using either best 5 or best 10 or best 20 companies......66
Figure15: Portfolio Management System using trend reversal strategy.............................67
Figure16: Comparison mean reversal strategy with normal strategy.................................68
viii
List of Tables
Table1. Composition of different Agents................................................................................26
Table2: Details of 10, FTSE 100 Companies..........................................................................32
Table3: Experimental results for 100 run on 5, FTSE100 companies.................................33
Table4: Setting Exploration rate.............................................................................................35
Table5: Combined Results for Different exploration rate....................................................36
Table6: Setting discount rate...................................................................................................36
Table7: Setting trace decay parameter...................................................................................37
Table8: Experimental Results for 3 learning methodology..................................................39
Table9: Setting Exploration Rate............................................................................................41
Table10: Setting discount rate.................................................................................................42
Table11: Setting trace decay....................................................................................................42
Table12: Comparative results for 90 FTSE 100 companies with delayed reward strategy
for single step reward only RL................................................................................................45
Table13: Results of 90 FTSE100 Companies for different RL with new delayed reward
strategy.......................................................................................................................................50
Table 14: Comparative Results for FTSE 100 companies using Passive Set......................54
Table15: Comparison with Only active set and with additional passive set approach......55
Table16: Combined Results single step RL without Passive set and multi step Q with
passive set...................................................................................................................................57
Table17: Portfolio Optimization using single best Agent.....................................................61
Table18: Monthly Portfolio management Using Best 3 agents.............................................64
Table19: Monthly Portfolio re-balancing using different Number of Best Agents............65
Table20: Portfolio Management System with quarterly re-balancing taking best 5
companies..................................................................................................................................65