i PRAISE FOR LEARNING ANALYTICS “An outstanding contribution to the field. A very well researched and engaging book filled with compelling research, sound theory and excellent examples to help practitioners move beyond traditional activity measures to far more impactful measures of effectiveness and impact. I strongly recommend this work to anyone new to the field as well as those who are seasoned practitioners.“ David Vance, Executive Director, Center for Talent Reporting “The second edition of Learning Analytics reaffirms the fundamentals of the first edition—that evaluation principles when standardized and enabled by technology—can provide insights about which programs are helping achieve business goals and which need improvement. This edition also brings a new methodology to the table, the Portfolio Evaluation Approach, that helps L&D organizations align and deeply connect with the business strategy. This edition shows how to measure informal learning, and finally, the authors address the broader category of HR analytics and how it can be tackled using many of the same principles that drive learning analytics. Read it. Use it. Improve your measurement practices.” Kimo Kippen, Founder of Aloha Learning Advisors and former Chief Learning Officer at Hilton Worldwide “Learning Analytics is the most thoroughly researched book on analytics that has ever come to market. It covers all the meaningful analytics models, explaining how they’ve been applied along with the pros and cons of each. The purpose of analytics is to link the effects of learning programs to organizational operations. The authors show only a small percentage of organizations venture beyond Level 2Learning. In my 30 years’ experience, I found training managers don’t attempt to show value added for two reasons. One, they don’t know how to do it. Two, they fear their programmes don’t add measurable value. If they reach the point where they want to analyse and report the effects of development programs, this book will guide them smoothly along that path.” Jac Fitz-enz, Founder, Saratoga Institute and Human Capital Source ii “If you need to evaluate your investments in talent, or use HR analytics to inform your business decisions, then this book is a godsend.” Doug Gray, CEO and Founder of Action Learning Associates and former President of ATD Nashville “Learning Analytics not only builds a strong science-based business case for ‘why learning analytics’ is needed, but goes beyond to identify the gap between what L&D is measuring in most organizations vs what CEOs want to be measured and achieved. Using a framework of efficiency, effectiveness and business outcome metrics and methods this book resolves the business to L&D gap, in a well-crafted step by step process. My favourite phrase in the book is ‘The future is for those who can predict it’. This book is a fantastic roadmap, guidebook and treasure trove of research, enabling L&D leaders and practitioners alike to solve that most obvious yet difficult of challenges, clearly linking L&D programmes to all areas of talent management in a cohesive, strategic way to positively impact performance and productivity.” Jeff Higgins, Founder and CEO, HCMI iii Learning Analytics Using talent data to improve business outcomes SECOND EDITION John R Mattox II Peggy Parskey Cristina Hall iv Publisher’s note Every possible effort has been made to ensure that the information contained in this book is accurate at the time of going to press, and the publishers and authors cannot accept responsibility for any errors or omissions, however caused. No responsibility for loss or damage occasioned to any person acting, or refraining from action, as a result of the material in this publication can be accepted by the publisher or the authors. First published in Great Britain and the United States in 2020 by Kogan Page Limited Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licences issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned addresses: 2nd Floor, 45 Gee Street 122 W 27th St, 10th Floor 4737/23 Ansari Road London New York, NY 10001 Daryaganj EC1V 3RS USA New Delhi 110002 United Kingdom India www.koganpage.com Kogan Page books are printed on paper from sustainable forests. © John R Mattox II, Peggy Parskey and Cristina Hall 2020 The right of John R Mattox II, Peggy Parskey and Cristina Hall to be identified as the authors of this work has been asserted by them in accordance with the Copyright, Designs and Patents Act 1988. ISBNs Hardback 9781789662986 Paperback 9781789663006 Ebook 9781789663013 British Library Cataloguing-in-Publication Data A CIP record for this book is available from the British Library. Library of Congress Control Number 2020932529 Typeset by Integra Print production managed by Jellyfish Printed and bound by CPI Group (UK) Ltd, Croydon CR0 4YY v Dedication To my family and colleagues, thank you for your support in this endeavour. John R Mattox, II My sincere thanks to colleagues and friends in the learning and measurement professions who have helped me expand my thinking and insights about this ever-changing field. My deepest thanks to my husband, Kevin, whose support on this effort and throughout the years has made it possible for me work on projects that give me joy. Peggy Parskey For Chris and Talon, and all who are passionate about maximizing human potential through lifelong learning. Cristina Hall vi THIS PAGE IS INTENTIONALLY LEFT BLANK vii CONTENTS List of figures and tables xii About the authors xviii Foreword xix How to read this book xxv Acknowledgements xxvii PART ONE The L&D value gap and how to close it 01 The rise of learning analytics 3 Why is all of this important? 6 Standards are coming 7 Data availability 8 Changing the way talent analytics work gets done 9 Providing unique insight into employee behaviour 12 Learning analytics rises 13 Notes 19 02 What is learning analytics? 21 Introduction 21 Learning analytics today: measure for measure, what should be measured? 23 Why measure learning? 25 Most organizations start with the simple: measure training activity and satisfaction 26 Efficiency, effectiveness and business outcomes: closing the learning measurement gap 28 The journey to learning analytics 29 The Four Levels of Evaluation 30 The Return on Investment Methodology 31 Impact Measurement Framework 33 Success Case Method 34 viii CONTENTS Performance-Based Evaluation 36 Conclusion 44 Notes 46 03 The value-centred learning organization: A new evaluation paradigm 49 Volume is not value 49 We’re already delivering value, though… right? 50 Delivering and demonstrating value: the Talent Development Value Framework 52 The Talent Development Value Framework in action 61 Advancing measurement maturity 63 Conclusion 65 Notes 66 PART TWO Establishing sound measurement practices 04 Aligning L&D’s value with the C-suite: The Four Value Drivers and Portfolio Evaluation 71 What the C-suite wants from L&D 71 Connecting L&D with the business strategy 73 The Four Value Drivers 74 Building business alignment 76 Translating value drivers to action: Portfolio Evaluation for L&D 78 Immediate benefits of portfolio alignment 83 Additional benefits: portfolio management 85 Change the conversation 90 Conclusion 91 Notes 91 05 Linking learning to business impact 93 What works? 93 Why does it work? 95 Experimental designs 99 Alternatives to experimental designs 103 Alternative designs: practical ways forward 103 CONTENTS ix Conclusion 114 Notes 116 06 The new leading indicators of success and how to manage them 118 Your training programmes may not be as good as you think they are 118 Scrap learning and how to reduce it 121 Performance improvement 126 Net promoter score 129 Manager support and how to improve it 131 Predictive Learning Impact Model 2.0: Causal Modelling 140 Conclusion 154 Notes 155 07 Developing a sustainable reporting strategy 156 The role of reporting in learning analytics 156 Getting started: design principles 159 Components of an effective reporting strategy 161 Reporting strategy development 170 Critical success factors 172 Perform gap assessment 175 Implementing the strategy 181 Special cases: dashboards and scorecards 183 Monitor the strategy: success indicators 188 Conclusion 189 Notes 190 08 Technology’s role in learning measurement 192 What should technology do? 193 Benefits and costs of learning technologies 194 The requirements for a new technology system in the BI space 200 The challenge of self-reported data 206 What is the ROI of technology systems? 209 Applying principles of business intelligence systems to L&D 210 Additional technologies 215 Conclusion 218 Notes 219