Using Approximate Dynamic Programming to Solve the Military Inventory Routing Problem with

123 Pages·2017·1.3 MB·English

Air Force Institute of Technology
AFIT Scholar
Theses and Dissertations Student Graduate Works
3-26-2015
Using Approximate Dynamic Programming to
Solve the Military Inventory Routing Problem with
Direct Delivery
Rebekah S. McKenna
Follow this and additional works at:https://scholar.afit.edu/etd
Recommended Citation
McKenna, Rebekah S., "Using Approximate Dynamic Programming to Solve the Military Inventory Routing Problem with Direct
Delivery" (2015).Theses and Dissertations. 123.
https://scholar.afit.edu/etd/123
This Thesis is brought to you for free and open access by the Student Graduate Works at AFIT Scholar. It has been accepted for inclusion in Theses and
Dissertations by an authorized administrator of AFIT Scholar. For more information, please [email protected].
Using Approximate Dynamic Programming to
Solve the Military Inventory Routing Problem
with Direct Delivery
THESIS
MARCH 2015
Rebekah S. McKenna, Second Lieutenant, USAF
AFIT-ENS-MS-15-M-140
DEPARTMENT OF THE AIR FORCE
AIR UNIVERSITY
AIR FORCE INSTITUTE OF TECHNOLOGY
Wright-Patterson Air Force Base, Ohio
DISTRIBUTION STATEMENT A
APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED.
The views expressed in this document are those of the author and do not reflect the
officialpolicyorpositionoftheUnitedStatesAirForce,theUnitedStatesDepartment
of Defense or the United States Government. This material is declared a work of the
U.S. Government and is not subject to copyright protection in the United States.
AFIT-ENS-MS-15-M-140
USING APPROXIMATE DYNAMIC PROGRAMMING TO SOLVE THE
MILITARY INVENTORY ROUTING PROBLEM WITH DIRECT DELIVERY
THESIS
Presented to the Faculty
Department of Operational Sciences
Graduate School of Engineering and Management
Air Force Institute of Technology
Air University
Air Education and Training Command
in Partial Fulfillment of the Requirements for the
Degree of Master of Science Opertaions Research
Rebekah S. McKenna, BS
Second Lieutenant, USAF
MARCH 2015
DISTRIBUTION STATEMENT A
APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED.
AFIT-ENS-MS-15-M-140
USING APPROXIMATE DYNAMIC PROGRAMMING TO SOLVE THE
MILITARY INVENTORY ROUTING PROBLEM WITH DIRECT DELIVERY
Rebekah S. McKenna, BS
Second Lieutenant, USAF
Committee Membership:
Lt Col Matthew J. Robbins, PhD
Chair
LTC Brian J. Lunday, PhD
Member
AFIT-ENS-MS-15-M-140
Abstract
The United States Army uses Vendor Managed Inventory (VMI) replenishment to
manage resupply operations while engaged in a combat environment; upper-echelon
organizations (e.g., a brigade) maintain situational awareness regarding the inven–
tory of lower-echelon organizations (e.g., battalions and companies). The Army is
interested in using a fleet of cargo unmanned aerial vehicles (CUAVs) to perform re–
supply operations. We formulate an infinite horizon, discrete time stochastic Markov
decision process model of the military inventory routing problem with direct delivery,
the objective of which is to determine an optimal unmanned tactical airlift policy
for the resupply of geographically dispersed brigade combat team elements operating
in an austere, Afghanistan-like combat situation. An approximate policy iteration
algorithm with Bellman error minimization using instrumental variables is applied to
determine near-optimal policies. Within the least-squares temporal differences policy
evaluation step, we use a modified version of the Bellman equation that is based on
the post-decision state variable. Computational results are obtained for examples
based on representative resupply situations experienced by the United States Army
in Afghanistan.
Key words: inventory routing problem, Markov decision process, approximate
dynamic programming
iv
For my family, especially my grandmothers. Their unwavering support was
invaluable.
v
Acknowledgements
I would like to express my sincere gratitude to my advisor, Dr. J.D. Robbins,
for his continuous patience, knowledge, and support during this effort. Without his
guidance, this endeavor would not have been possible.
Rebekah S. McKenna
vi
Table of Contents
Page
Abstract ............................................................... iv
Dedication ..............................................................v
Acknowledgements ...................................................... vi
List of Figures .......................................................... ix
List of Tables............................................................x
I. Introduction ........................................................1
II. Literature Review ...................................................5
2.1 Inventory Routing Problem .......................................5
2.2 The Military Inventory Routing Problem ...........................9
2.3 Markov Decision Processes.......................................12
III. Methodology ......................................................18
3.1 Problem Description ............................................18
3.2 MDP Formulation ..............................................19
3.3 Exact Solution .................................................25
3.4 Approximate Solution...........................................26
IV. Computational Example.............................................33
4.1 MDP Parameterization..........................................33
4.2 2-COP Problem Instance ........................................35
4.3 3-COP Problem Instance ........................................45
4.4 Investigation of the Stock-out Penalty, τ ..........................50
4.5 12-COP Problem Instance .......................................54
4.6 Experimental Design............................................56
4.7 Results and Analysis............................................59
V. Conclusions and Recommendations ...................................75
5.1 Conclusions....................................................75
5.2 Limitations ....................................................77
5.3 Future Research................................................78
Appendix A. Acronyms .................................................79
Appendix B. Computational Example: 2-COP .............................80
vii
Page
Appendix C. Computational Example: 3-COP .............................94
Appendix D. Storyboard ...............................................106
Bibliography ..........................................................107
viii