Table Of ContentAction Evaluation of
Health Progran,n,es
and Changes
A handbook for a
user-focused approach
John Svretveit
Foreword by
Chris Ham
Boca Raton London New York
CRC Press is an imprint of the
Taylor & Francis Group, an informa business
First published 2002 by Radcliffe Publishing
Published 2018 by CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2002 John 0vretveit
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
ISBN-13: 978-1-85775-925-9 (pbk)
This book contains information obtained from authentic and highly regarded
sources. While all reasonable efforts have been made to publish reliable data and
information, neither the author[s] nor the publisher can accept any legal
responsibility or liability for any errors or omissions that may be made. The
publishers wish to make clear that any views or opinions expressed in this book
by individual editors, authors or contributors are personal to them and do not
necessarily reflect the views/opinions of the publishers. The information or
guidance contained in this book is intended for use by medical, scientific or
health-care professionals and is provided strictly as a supplement to the medical
or other professional's own judgement, their knowledge of the patient's medical
history, relevant manufacturer's instructions and the appropriate best practice
guidelines. Because of the rapid advances in medical science, any information or
advice on dosages, procedures or diagnoses should be independently verified. The
reader is strongly urged to consult the relevant national drug formulary and the
drug companies' and device or material manufacturers' printed instructions, and
their websites, before administering or utilizing any of the drugs, devices or
materials mentioned in this book. This book does not indicate whether a
particular treatment is appropriate or suitable for a particular individual.
Ultimately it is the sole responsibility of the medical professional to make his or
her own professional judgements, so as to advise and treat patients appropriately.
The authors and publishers have also attempted to trace the copyright holders of
all material reproduced in this publication and apologize to copyright holders if
permission to publish in this form has not been obtained. If any copyright
material has not been acknowledged please write and let us know so we may
rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be
reprinted, reproduced, transmitted, or utilized in any form by any electronic,
mechanical, or other means, now known or hereafter invented, including
photocopying, microfilming, and recording, or in any information storage or
retrieval system, without written permission from the publishers.
Trademark Notice: Product or corporate names may be trademarks or registered
trademarks, and are used only for identification and explanation without intent
to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http:// www.crcpress.com
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library.
Typeset by Joshua Associates Ltd, Oxford
Contents
Foreword vii
Preface viii
About the author ix
1 Action evaluation: what it is and why evaluate? 1
Introduction 1
Action evaluation for evaluating health
programmes and changes 4
Outline of the book 8
Why evaluatel 11
What is evaluationl 12
Types of evaluation 15
What evaluation can and cannot do 16
Summary 17
Part 1 How to make an evaluation 19
2 How to evaluate a health service or programme 21
Introduction 21
Step 1: preparation 22
Step 2: design planning 26
Step 3: design choice 31
Summary: how to carry out a service or programme
evaluation 34
Data collection for the ‘six criteria' method 37
Summary 40
3 How to evaluate a policy, change or reform to
healthcare 41
Introduction 41
Reform is an intentional change to organisation 43
Step 1: define the change 45
Step 2: define the evaluation perspective 47
Step 3: define the design 48
Step 4: gather data 50
Step 5: consider explanations 52
IV Contents
Step 6: write and communicate evaluation
conclusions 53
Summary 54
4 How to plan an evaluation 57
Introduction 57
Evaluation feasibility assessment (EFA) 58
The 'quick' guide 60
The 'nine questions' guide 61
Summary 69
5 Practical issues and planning 71
Introduction 71
Practical problems to expect and minimise 72
Phases in an evaluation: who does whatl 79
Summary 83
Part 2 Evaluation tools 85
6 Concepts for making sense of an evaluation 87
Introduction 87
Concept 1: the evaluation 'user' or 'customer' 89
Concept 2: value criteria 90
Concept 3: intervention 92
Concept 4: target 95
Concept 5: outcome 96
Concept 6: objectives 97
Concept 7: confounder 98
Concept 8: control 99
Concept 9: data gathering 100
Summary 101
7 Evaluation design 103
Introduction 103
Design: time perspective 104
The 'box model' for drawing an evaluation design 107
Overview of the six designs 109
The 'NIPOO' model 110
Summary 111
8 Process evaluation designs 113
Introduction 113
Type 1: descriptive 113
Type 2: audit 115
Summary 118
Contents V
9 Outcome evaluation designs 119
Introduction 119
Type 3: single before-and-after 119
Type 4: comparative 121
Type 5: randomised controlled trial 123
Type 6a: intervention to a health organisation -
impact on personnel 126
Type 6b: intervention to a service - impact on
patients 127
Summary 128
10 Assessing an evaluation and choosing a design 129
Introduction 129
How to assess an evaluation: the *4D’ method 129
How to choose a design 134
Initial questions for analysing or planning an
evaluation 138
Summary 139
Part 3 Specific subjects 141
11 Data-gathering methods 143
Introduction 143
Using existing data sources 146
Observation 148
Interviewing 150
Surveying and questionnaires 154
Measurement methods 156
Analysing data 159
Summary 163
12 The politics and ethics of evaluation 165
Introduction 165
The politics of evaluation 166
Ethical evaluation 173
Summary 176
13 Evaluating quality and qualefficiency 179
Introduction 179
Why evaluate qualityί 181
Defining and measuring quality 184
Evaluating service quality 188
Evaluating a quality system or quality programme 190
VI Contents
Evaluating qualefftciency: an integrated approach
to performance assessment 193
Practical steps for a quality evaluation 196
Summary 198
14 Evaluation paradigms and perspectives 199
Introduction 199
Positivist and phenomenological paradigms 200
Types of evaluation 203
Evaluation approaches 205
Summary 210
References 213
Appendix 1 Definitions 219
Appendix 2 Rehabilitation example design 229
Appendix 3 Questions for informants 231
Appendix 4 Data-gathering plan 233
Appendix 5 Data for health programme evaluation 235
Appendix 6 Step objectives model 239
Index 241
Foreword
John 0vretveit has written a clear and practical introduction to
evaluation in health services. Drawing on his experience as a
researcher and teacher, 0vretveit focuses particularly on evalua
tions intended to influence policy and practice. In a wide-
ranging review, he explains how evaluations are conducted,
reviews the tools available to evaluators, and discusses the
political and ethical dilemmas that may arise.
The great virtue of 0vretveit's approach is that it is simple
without being simplistic. 0vretveit recognises both the oppor
tunities available to evaluators and the obstacles they may
encounter along the way. The book will therefore be a valuable
introduction for evaluators and those involved in commission
ing evaluations.
As its subtitle indicates, Action Evaluation of Health Pro
grammes and Changes is a handbook that takes the reader on a
step-by-step guide through the field. It will be of particular
interest to those involved in evaluations in which experimental
designs are not feasible and where alternative methods have to
be adopted. 0vretveit helpfully sets out the strengths and
weaknesses of different approaches and includes in the text a
series of practical tips based on his experience. At a time when
'evidence based everything' is high on the agenda, his introduc
tion to user focused evaluation will be welcomed by researchers
and practitioners.
Chris Ham
Health Services Management Centre
University of Birmingham
October 2001
Preface
The need for evaluation in healthcare has never been greater, yet
the evaluation methods used often do not provide the evidence
which decision makers need. Action evaluation focuses on the
evaluation user's needs and designs the evaluation to answer
their questions within the resources and time available. The
approach can be used to maximise the usefulness of an evalu
ation carried out using either traditional methods or methods
less well known in healthcare.
This book describes this 'user-focused' action evaluation
approach for evaluating health programmes, policies and
changes. It shows that evidence for making decisions about
these interventions can be gathered using a broader range of
methods than is usually recognised in healthcare. In asking
'evaluation for whom and for what?', the book defines validity
of evidence in relation to usefulness as well as in terms of
whether systematic methods were followed.
Managers, practitioners, policy makers, researchers and stu
dents will find this a practical book on evaluation which gives a
broad view of the approaches needed. It gives tools to make
sense of an evaluation quickly, as well as guidance for carrying
one out, and for ensuring that evaluators gather evidence which
can be used for making better informed decisions.
John 0vretveit
October 2001
About the author
Dr John 0vretveit is an experienced evaluator of change and
teacher of both social science and medical research evaluation
methods. He is Professor of Health Policy and Management at
the Nordic School of Public Health in Gothenberg, the School of
Medicine, University of Bergen, and the Karolinska Institute,
Stockholm.
