Information Technology Audit: Systems Alignment and Effectiveness

314 Pages·2009·2.78 MB·English

Information Technology Audit: Systems
Alignment and Effectiveness Measures

Mathew Nicho
B.Sc., MBA, M.BUS (IT)

a thesis submitted to the graduate faculty of design and creative technologies
AUT University
in partial fulfilment of the
requirements for the degree of
doctor of philosophy

School of Computing and Mathematical Sciences

Auckland, New Zealand
2008
ii
Declaration

I hereby declare that this submission is my own work and that, to the best of
my knowledge and belief, it contains no material previously published or
written by another person nor material which to a substantial extent has been
accepted for the qualification of any other degree or diploma of a University or
other institution of higher learning, except where due acknowledgement is
made in the acknowledgements.

...........................
Signature
iii
Acknowledgements

The researcher is grateful to the following people whose contributions are
invaluable and without whom the thesis might not have been a reality. Others
including friends and colleagues have provided motivation and inspiration to
get started and to complete the task. They are all acknowledged with gratitude.
From an academic point of view, first of all the researcher wishes to
thank the supervisor, Dr Brian Cusack, for the constant monitoring, support,
encouragement and advice that was provided in the last four years to guide the
completion of the thesis. The contribution of the supervisor will be a long-term
asset in moulding the outlook of the researcher to reach a higher academic
level. Secondly the researcher would like to thank the second supervisor Dr.
Stephen McDonnell who always takes time to evaluate the different pieces of
work and give constructive criticisms. Thirdly the researcher is thankful to Dr.
Narayan Ramasubbu of the School of Information Systems, Singapore
Management University who had not only properly guided the researcher
during his research work in Singapore, but also helped in finding the
approppriate organisations to do the study. Also a million thank are due to the
respondents of the six organisations in New Zealand and Singapore who, in
spite of their busy work schedule was kind enough to evaluate the model and
give their valuable comments. The three experts who took their time to go
through the hard copy of the model and fill up the GQM template require
special mention.
I wish to extend my sincere gratitude and appreciation to my wife, who
had been with me through all the trials and tribulations, taking great sacrifices
physically, mentally and financially to see the completion of my PhD. My ten
year old son Shaun and my four year old daughter Sasha had been a constant
source of constructive distraction from my work without whom, I would have
gone insane. I would also like to thank my mother Leena who had prayed
fervently to see the successful completion of my PhD. My PhD friends and
colleagues at the School of Computing and Mathematical Sciences of AUT had
iv
been with me throughout these years and whose encouragement, support, and
constructive academic discussions have helped me in having a positive outlook
of this research.
I am also thankful to the staff of School of Computing and
Mathematical Sciences who have rendered valuable assistance in times of need
especially Dr. Albert Yep who approved my trip to Hawaii to present the
critical paper, Celia who had been tormented by my various requests but have
been patient enough to go through the proper administrative process for the
various requests of the doctorate process, and the staff who I have met at the
corridors of AUT who had encouraged me throughout the 4 years of PhD. The
assistance of AUT administrators, including the AUT Ethics Committee is also
acknowledged with appreciation. Numerous people have helped the researcher
in getting the contacts and entry into professional networks for field data
collection and are acknowledged. Staff at AUT, in SCIS, EC, friends and
colleagues who helped in these matters, are all acknowledged with gratitude.
v
Abstract
Information technology audit has proven to be a relatively new, less researched
and rapidly expanding field among large, medium and even small businesses
(commercial and non-commercial organisations). The implementation rate has
grown rapidly and presents a huge growth market for audit consultants due to
the need for transparency and compliance with regulation (for example:
Sarbanes Oxley Act) and the need to be competitive in the marketplace. The
audit process is being conducted mainly by consultants following a traditional
process but using different proprietary approaches and mostly done manually.
The purpose of this study is to present a scientific method to attach a purely
measurement focus to the auditing process so as to provide an auditing as well
as a quantitative outcome of the performance to the various IS entities that are
audited using a novel automated method that can save organisations
considerable resources in terms of time, cost and effort.
The nature of the topic directed the researcher to three domains of
information system (IS) namely studies on IS measurement, IT governance and
software engineering. These areas provided information on the nature of IS
measurement and the models used; the process of auditing/measurement and
the corresponding frameworks used; the principles and methodology of
measurement of IS entities; and measurement models used both in the software
engineering and information systems domain. The review of the literature gave
rise to the research question and the COBIT-GQM (Control Objectives for
Information Technology Audit) – Goal Question Metrics) model. The research
question that had emerged out of the four propositions “How can an IT audit or
governance framework be used to measure the effectiveness of IS entities in a
scientific manner using customised and goal oriented metrics” along with the
nature of data sought (positivist), guided the researcher to qualitative research
using multiple case studies to test the theoretical model (grounded theory) that
had emerged out of the literature review.
vi
The theoretical model was automated (with a front end interface and a back-
end database) and initially tested for usability issues. Then the common COBIT
control objective that was obtained through an initial survey was entered into
the database along with a set of questions and metrics (developed by the
researcher by following the given GQM guidelines). This application that was
demonstrated, and given for evaluation in four organisations gave rise to
expected and surprising results. While the respondents expressed their desire to
incorporate a customised and goal oriented measurement perspective to their IT
audit/performance functions, that would save them time, effort and cost,
numerous suggestions were provided that need to be incorporated into the
model to make it fully functional. Notable among them are the need to embed a
multiple contextual qualifying layer, incorporating benchmarking feature to the
model, and the need to link this with the maturity model. These were
incorporated into the model and a comprehensive model incorporating all the
suggestions was created.
The qualitative case study method being used here more to evaluate a
theory, provided a sound base for future studies to generate hypothesis that can
be evaluated using quantitative survey methods for the model to be generalised.
IT auditing being a relatively new, less researched, conventional and high
growth oriented field, the use of an innovative, comprehensive, automated and
scientific method of audit and measurement method will satisfy the implied
need for organisations to incorporate the diverse audit/measurement/
control/standards into one comprehensive method and this research is a major
step in this direction. Since the new model is comprehensive and can be
automated organisations can economise in terms of time, cost and effort.
Irrespective of the nature of economic cycle the need for economising in terms
of cost, time and effort is universal for all organisations.
vii
Table of Contents
Declaration .............................................................................................................. ii
Acknowledgement .................................................................................................. iii
Abstract ................................................................................................................... v
Table of Contents .................................................................................................. vii
Appendices ......................................................................................................... xviii
List of Tables ......................................................................................................... xv
List of Figures ....................................................................................................... xx
Abbreviations ...................................................................................................... xxii

Chapter – 1 Introduction
1.0 Introduction ................................................................................................................... 1
1.1 Studies on IS Measurement .......................................................................................... 2
1.2 Gaps in the Relevant Knowledge Areas ....................................................................... 3
1.3 Operationalising the Research ...................................................................................... 3
1.4 Expected Research Outcomes ....................................................................................... 4
1.5 Positioning of the Study ................................................................................................ 5
1.6 Structure of the thesis ................................................................................................... 6
1.7 Conclusion .................................................................................................................... 7

Chapter – 2 Literature Review
2.0 Introduction ....................................................................................................................8
2.1 Measurement of IS Effectiveness ..................................................................................9
2.1.1 The Need for Measuring IS Effectiveness........................................................... 9
2.1.1.1 Measurement Relevance – An IS Perspective ......................................... 10
2.1.1.2 Measurement Relevance – An SE Perspective ........................................ 11
2.1.1.3 Key Issues in IS ....................................................................................... 11
2.1.1.4 Critical Success factors in IS Success ...................................................... 13
2.1.2 Challenges of Measuring IS Effectiveness ........................................................ 14
2.1.3 Perspectives of Research on IS Effectiveness ................................................... 15
2.1.3.1Unidimensional Nature of IS Measurement Studies ................................. 15
2.1.3.2Broad Studies on IS Measurement ............................................................ 16
2.1.3.3Dimensions of IS Success Measurement .................................................. 16
viii
2.1.4 Measurement Principles – An IS Perspective ................................................... 17
2.1.4.1Dimensions of IS Success ......................................................................... 18
2.1.4.2Functional Measurement of IS .................................................................. 18
2.1.4.3Objective and Subjective Measurement ................................................... 19
2.1.4.4Use of Measures/Metrics/Scales ............................................................... 20
2.1.4.5Performance Oriented Measurement ........................................................ 20
2.1.5 Models Evaluation ............................................................................................. 21
2.1.6 Overlap of ITG/Audit Concepts with IS Measurement ..................................... 23
2.1.6.1Key Issues in IS – ITG Perspective .......................................................... 24
2.2 IT Governance and Measurement .............................................................................. 25
2.2.1 Measurement in IT Governance ........................................................................ 25
2.2.2.1 Systems Alignment and Effectiveness Measures .................................... 28
2.2.2 An Evaluation of IT Control/Audit Frameworks .............................................. 29
2.2.2.1 The COBIT IV Framework ...................................................................... 31
2.2.2.2 Mapping of ITG Domain with COBIT .................................................... 33
2.2.2.3 Mapping of ITG and COBIT Focus Areas .............................................. 34
2.2.3 Measurement in COBIT .................................................................................... 34
2.2.3.1 Measurement Tools in COBIT ................................................................ 35
2.2.3.1.1 Issues in Measurement using COBIT KPI and KGI ................ 35
2.2.3.2 Measurement Models in COBIT .............................................................. 36
2.2.3.2.1 Maturity Models in COBIT ...................................................... 36
2.2.3.2.2 The Balanced Score Card ......................................................... 37
2.2.3.3 Issues in COBIT....................................................................................... 38
2.2.4 Alignment of Metrics with Goals, COs, and Control Process ........................... 39
2.2.5 COBIT as a Measurement Process Framework................................................. 41
2.2.5.1 Mapping of COBIT with the Measurement Model of Ashley ................. 41
2.2.5.2 Mapping of COBIT with the Measurement Process of Offen & Jeffrey . 45
2.3 Measurement in Software Engineering ....................................................................... 47
2.3.1Measurement Principles in Software Engineering ............................................. 47
2.3.1.1 Metrics ..................................................................................................... 48
2.3.1.2 The Object of Measurement .................................................................... 49
2.3.2 Application of Software Metrics to the IS Domain ........................................... 50
2.3.3 Relevance of Measurement in Software Engineering ....................................... 52
2.3.4 Challenges in Software Measurement ............................................................... 54
ix
2.3.5 Metrics Generation Models ............................................................................... 56
2.3.5.1 The GQM Model ..................................................................................... 57
2.3.5.1.1 Critical Evaluation of the GQM Model .................................... 58
2.3.5.1.2 The GQM Approach ................................................................ 59
2.4 Integrating GQM into COBIT .................................................................................... 61
2.4.1 The COBIT-GQM Model .................................................................................. 61
2.4.1.1 Measuring COBIT Using IT Goals in Lieu of the DCO .......................... 64
2.4.2 A Theoretical Demonstration with an Example ................................................ 64
2.4.3 Metrics ............................................................................................................... 69
2.4.4 Model Automation ............................................................................................ 69
2.5 Propositions ................................................................................................................ 71
2.6 Conclusion .................................................................................................................. 72
Chapter – 3 Research Methodology
3.0 Introduction ................................................................................................................. 74
3.1 The Research Question ............................................................................................... 75
3.2 Research Philosophy ................................................................................................... 81
3.2.1 Research Approach ..................................................................................... 83
3.2.2 Research Paradigm ..................................................................................... 84
3.3 Research Design ......................................................................................................... 86
3.3.1 Steps in the Research Process ..................................................................... 86
3.3.2 The Model Followed ................................................................................... 88
3.4 Methodology Review of Previous Research ............................................................... 89
3.4.1 Case Study 1 ............................................................................................... 90
3.4.2 Case Study 2 .............................................................................................. 90
3.4.2 Case Study 3 .............................................................................................. 91
3.5 Research Methods ....................................................................................................... 93
3.5.1 Case Study Method ..................................................................................... 94
3.5.1.1 Multiple Case Study ................................................................... 96
3.5.2 Data Collection Techniques ........................................................................ 97
3.5.3 Criteria for the Selection of the Organisation ............................................. 97
3.5.4 Usability Study ........................................................................................... 99
3.5.5 Sources of Data ........................................................................................... 99
3.5.6 Data Collection ........................................................................................... 99
x
3.5.6.1 Data Types ................................................................................ 100
3.5.6.2 Data Collection Process - Steps ................................................ 100
3.5.6.3. Nature of Data .......................................................................... 100
3.5.7 Location of the Study ................................................................................ 102
3.5.8 Processing of Data .................................................................................... 103
3.5.9 Reliability and Validity ............................................................................. 104
3.6 Analysis of Data........................................................................................................ 105
3.6.1 Detailed Plan of the Analysis and Discussion .......................................... 107
3.6.1.1 Tidying up ................................................................................. 107
3.6.1.2 Finding Items ............................................................................ 108
3.6.1.3 Creating Stable Sets of Items .................................................... 108
3.6.1.4 Creating Patterns ....................................................................... 110
3.6.1.5 Assembling Structures .............................................................. 110
3.6.2 Reporting Case Studies ............................................................................. 111
3.7 Problems Expected to be Encountered ..................................................................... 112
3.8 Conclusion ................................................................................................................ 113
Chapter - 4 Analysis of the Findings
4.0 Introduction ................................................................................................................114
4.1 Case Profile ............................................................................................................... 115
4.1.1 Case NZ 1 ................................................................................................. 116
4.1.2 Case NZ 2 ................................................................................................. 117
4.1.3 Case NZ 3 ................................................................................................. 117
4.1.4 Case SG 1 ................................................................................................. 118
4.2 Analysis of the Cases ............................................................................................... 118
4.2.1 Tidying up (Stage - 1) ............................................................................... 118
4.2.1.1 Definition of Nodes .................................................................. 119
4.2.1.2 Coding Summary - NZ 1 .......................................................... 123
4.2.1.3 Coding Summary - NZ 2 .......................................................... 124
4.2.1.4 Coding Summary - NZ 3 .......................................................... 125
4.2.1.5 Coding Summary - SG 1 ........................................................... 127
4.2.2 Finding Items (Stage - 2) ......................................................................... 128
4.2.2.1 NZ 1 (Stage - 2) ........................................................................ 130
4.2.2.1.1 Functionality ............................................................. 133