Research Impact: Guidance on Advancement, Achievement and Assessment

145 Pages·2020·2.19 MB·English

by  Hugh P. McKenna

RRRRRRReeeeeeessssssseeeeeeeaaaaaaarrrrrrrccccccchhhhhhh IIIIIIImmmmmmmpppppppaaaaaaacccccccttttttt
Guidance on Advancement,
Achievement and Assessment
Hugh P. McKenna
123
Research Impact
Hugh P. McKenna
Research Impact
Guidance on Advancement,
Achievement and Assessment
Hugh P. McKenna
Ulster University
Ulster
UK
ISBN 978-3-030-57027-9 ISBN 978-3-030-57028-6 (eBook)
https://doi.org/10.1007/978-3-030-57028-6
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Switzerland AG 2021
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher,
whether the whole or part of the material is concerned, specifically the rights of translation,
reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any
other physical way, and transmission or information storage and retrieval, electronic adaptation,
computer software, or by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are
exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in
this book are believed to be true and accurate at the date of publication. Neither the publisher nor
the authors or the editors give a warranty, expressed or implied, with respect to the material
contained herein or for any errors or omissions that may have been made. The publisher remains
neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
I would like to acknowledge the unstinting support of my wife
Tricia, my son Gowain and my daughter Saoirse. My gratitude
goes to our Border Collie ‘Cooper’ who took me for long walks
when writer’s block arrived.
Preface
No researcher wants to see the results of their investigations languishing on
library shelves or being unknown by their peers and those who would benefit
from the findings. Rather, they want to see their research having impact on soci-
ety, the economy, culture, health or quality of life. Furthermore, high-q uality
research cannot be done without funding, and more and more funding bodies are
focusing their grants on research that is potentially impactful. In addition, for
almost 30 years, publicly funded research in the UK has been assessed for qual-
ity in a series of research assessment exercises. The most recent one is called the
Research Excellence Framework (HEFCE, 2014). As a result of the many
reviews of research impact generally and the REF 2014 specifically, this infor-
mation is in the public domain, and suitable references will be included as sign-
posts for those readers who seek more in-depth information.
Such exercises are also carried out in many countries across the world.
These include Finland, Norway, Sweden, Denmark, Holland, Italy, New
Zealand, Australia, Romania, Hong Kong, Germany and most recently the
Czech Republic. The results are mainly used to inform the allocation of pub-
lic funding for research and provide accountability for tax payers’ money.
Increasingly, these exercises are including the assessment of research impact
because governments believe it is not unreasonable to ask those whose
research work is undertaken at public expense to account for and provide
some evidence of their impact.
However, the traditional approach to assuring and assessing the quality of
university research has been through quantitative measures such as publica-
tions, citations, Ph.D. completions and research income. These are deeply
incorporated in the recruitment, retention and reward system for academic
staff. It is a truism that research that is published in the top journals and
highly cited has benefits for individual researchers but is less beneficial for
society. As research impact becomes more important globally, the traditional
metrics are beginning to be regarded as only providing a partial picture of full
academic impact.
Furthermore, the lessons learned over the last decade in the UK are appli-
cable and transferable to other countries. These are dealt with in this textbook
and include how to engage with external stakeholders to ensure the creation
of impact, how universities prepare for the assessment of their research
impact and how expert panel assessors actually review research impact. The
core of research impact assessment is the case study, and a substantial part of
Chap. 8 is devoted on how these should be written.
vii
viii Preface
The generation of research impact is complex and resource intensive. This
textbook explores these challenges in detail and where possible proposes
solutions. There is also a strong link between evidence-informed policy and
practice and research impact. This link is analysed and clarification provided
on the way forward. Perhaps one of the most difficult ways to create research
impact is through public engagement and networking with policy makers.
These are highlighted and suggestions for improvement proffered. An over-
view of how research impact is evaluated in other countries is also presented
with lessons that can be disseminated globally.
The research impact agenda is with us and is not going to go away. In fact,
it will play a greater role in further research assessment exercises and in
research grant applications. It encourages researchers to think more about
maximising the benefits of their research, and no reasonable person would
deny that this is a good thing. This is the reason why it has been embraced by
most academics, universities and governments. I predict that in the years to
come more countries will be including research impact assessment in their
review of research quality.
I would like to acknowledge the Research Impact Manager and Research
Impact Officers at Ulster University. Furthermore, I am grateful to all those
workshop delegates whose perceptive questions over the years have had the
desired ripple effect in developing my thinking about research impact. My
gratitude also goes to Karthik Periyasamy and Nathalie L’Horset-Poulain
from Springer who have been unstinting in their positive support and thought-
ful advice. Finally, my thanks to members of the REF team at HEFCE and
Research England who stimulated my interest in this topic over the past
decade.
In all the various international research evaluation exercises, different dis-
ciplines and expert panel may have subtle variations in how assessment is
undertaken. This textbook takes a general approach; it is highly recommended
that those who want more granular details refer to the official guidance.
Finally, I want to state that apart from the sources referenced, all the opinions,
views and assertions are mine alone. This also includes any errors in content
and stylistic inaccuracies. I am not representing Research England or any
other government body regionally, nationally or internationally. The next
REF has been severely affected by the Covid-19 pandemic. This has affected
key deadline dates regarding the submission and assessment of research
impact; these new dates have been included in Chap. 2.
Ulster, UK Hugh P. McKenna
Contents
1 Research Impact: The What, Why, When and How . . . . . . . . . . 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Reactive Approach to Research Impact . . . . . . . . . . . . 2
1.1.2 Active Approach to Research Impact . . . . . . . . . . . . . . 2
1.1.3 Proactive Approach to Research Impact . . . . . . . . . . . 2
1.2 Why Is Research Impact Important? . . . . . . . . . . . . . . . . . . . . 4
1.2.1 Moral Imperative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 The Role of Universities as Drivers for Impact . . . . . . 5
1.2.3 Research Impact as a Dictate from
Funding Bodies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.4 The Research Excellence Framework . . . . . . . . . . . . . 8
1.3 The REF and Research Impact . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Positive Views of Research Impact in the REF . . . . . . . . . . . . 11
1.5 Research Impact: Its Reach and Significance . . . . . . . . . . . . . 11
1.6 The REF Expert Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2 Research Impact: Drafting Case Studies . . . . . . . . . . . . . . . . . . . 17
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 The REF Impact Case Study Template . . . . . . . . . . . . . . . . . . 18
2.2.1 Stylistic Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3 Research Impact: How to Prepare and Submit
Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2 Structures and Processes Employed by Universities
for Research Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 Level 1: The Unit of Assessment . . . . . . . . . . . . . . . . . . . . . . . 34
3.4 Level 2: The Faculty Level . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.5 Level 3: The University Level . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.6 The Research Impact Manager . . . . . . . . . . . . . . . . . . . . . . . . 39
3.6.1 Key Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.6.2 Key Duties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.7 Commercial Tools for Tracking Research Impact . . . . . . . . . . 40
3.7.1 Vertigo Ventures-Impact Tracker . . . . . . . . . . . . . . . . . 40
ix
x Contents
3.7.2 Researchfish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.7.3 PURE Impact Tracker . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.8 Incorporating Impact Generation into
an Institution’s Research Environment . . . . . . . . . . . . . . . . . . 43
3.8.1 Civic University . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.8.2 Red Brick Universities . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.9 REF: Research Impact and Research Environment . . . . . . . . . 44
3.9.1 Institution Environment Statement (REF 5a) . . . . . . . . 45
3.9.2 UoA Environment Statement (REF 5b) . . . . . . . . . . . . 45
3.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4 Research Impact: How Assessors Prepare and Assess . . . . . . . . . 49
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.1.1 The Appointments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.1.2 The Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.1.3 The Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 What Makes Expert Panel Assessors Rate Impacts
Outstanding or Weak? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3 Expert Panel Assessors and Impact Confidentiality . . . . . . . . . 54
4.4 The Views of Assessors and Others
on REF Guidance for Impact Case Studies . . . . . . . . . . . . . . . 55
4.4.1 Views of REF Impact Assessors
on the REF Guidance for Impact Case Studies . . . . . . 55
4.4.2 Views of University Researchers
on the REF Guidance for Impact Case Studies . . . . . . 56
4.5 How Quantitative and Qualitative Data Help
Assessors Make Judgements? . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.5.1 How Quantitative and Qualitative Data
Assists the Assessment of Impact? . . . . . . . . . . . . . . . 58
4.6 Calculating a Quality Profile . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5 Research Impact: The Challenges . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.2 The Attribution Time Lag . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.3 Disciplinary Issues and Research Impact . . . . . . . . . . . . . . . . 63
5.4 A Resource Hungry Process . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.5 The Nature of Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.6 A Distraction from Doing Research . . . . . . . . . . . . . . . . . . . . 65
5.7 Political Policy and REF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.8 The Difficulty in Capturing Impact Comprehensively . . . . . . 66
5.9 Impact and the University Mission . . . . . . . . . . . . . . . . . . . . . 67
5.10 New Schools and Departments Being Disadvantaged . . . . . . . 68
5.11 Research Impact: Not Always Good . . . . . . . . . . . . . . . . . . . . 69
5.12 Conflicting Research Advice and Resultant Impacts . . . . . . . . 70
5.13 Gender Bias and Research Impact . . . . . . . . . . . . . . . . . . . . . . 72
5.14 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Contents xi
6 Research Impact: Engaging with the Public
and Policymakers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6.2 Engagement and Impact: The Relationship . . . . . . . . . . . . . . . 76
6.3 Research Advice on Covid-19 and Its Effect on Public
Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.4 Bodies that Assist Public Engagement . . . . . . . . . . . . . . . . . . . 79
6.5 Social Media and Altmetrics as Tools for Impact . . . . . . . . . . 80
6.6 Research Impact and Policymaking . . . . . . . . . . . . . . . . . . . . . 80
6.7 The Ten Commandments for Influencing Policymakers . . . . . 82
6.7.1 Get to Know the People Who Really Matter . . . . . . . . 82
6.7.2 Do not Just Criticise, Offer Solutions . . . . . . . . . . . . . 83
6.7.3 Give Policymakers a Strong Narrative . . . . . . . . . . . . . 83
6.7.4 Use Social Media Sparingly. . . . . . . . . . . . . . . . . . . . . 83
6.7.5 Use Ministerial Priorities as a Hook . . . . . . . . . . . . . . 83
6.7.6 Keep Politics Out of It . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.7.7 When Ministers Change, Start Again. . . . . . . . . . . . . . 84
6.7.8 Treat Returning Ministers Like New Ministers . . . . . . 84
6.7.9 Do Not Overlook the Power of Constituency MPs . . . 84
6.7.10 Above All, Be Consistent . . . . . . . . . . . . . . . . . . . . . . . 84
6.8 A Communications Toolkit for Healthcare Impact . . . . . . . . . 84
6.9 Research Methods the Encourage Impact . . . . . . . . . . . . . . . . 86
6.10 Open Science and Engagement for Impact . . . . . . . . . . . . . . . 87
6.11 Examples of Public Engagement’s Impact on Policy
and Society . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7 Research Impact: The Link to Evidence-Informed Practice . . . 91
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
7.2 Some Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
7.3 The Drive for Certainty and Agreement
in Research Impact? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.4 Research Impact: When Is Best Evidence Ignored? . . . . . . . . 98
7.5 Researchers: Their Role in Encouraging
or Discouraging Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
8 Research Impact: An Analysis of Approaches
in Health Sciences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8.2 An Analysis of Health Science Impacts Submitted
in the 2014 REF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
8.3 A Comparison of Research Impact Quality with Other
Components in REF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
8.3.1 A Historical Exploration of Research Impact
and Related Publications . . . . . . . . . . . . . . . . . . . . . . . 111
8.4 Impact Case Studies for REF2014 Relating to Health Care . . . 112