Demystifying Scholarly Metrics: A Practical Guide PDF
Preview Demystifying Scholarly Metrics: A Practical Guide
Demystifying Scholarly Metrics This page intentionally left blank DEMYSTIFYING SCHOLARLY METRICS A Practical Guide Marc W. Vinyard and Jaimie Beth Colvin Copyright © 2022 by Marc W. Vinyard and Jaimie Beth Colvin All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, except for the inclusion of brief quotations in a review, without prior permission in writing from the publisher. Library of Congress Cataloging-in-Publication Data Names: Vinyard, Marc W., author. | Colvin, Jaimie Beth, author. Title: Demystifying scholarly metrics : a practical guide / Marc W. Vinyard and Jaimie Beth Colvin. Description: Santa Barbara, California : Libraries Unlimited, [2022] | Includes bibliographical references and index. Identifiers: LCCN 2021028546 (print) | LCCN 2021028547 (ebook) | ISBN 9781440875939 (paperback) | ISBN 9781440875946 (ebook) Subjects: LCSH: Bibliometrics—Handbooks, manuals, etc. | Scholarly publishing—Evaluation. | Research—Evaluation—Statistical methods. Classification: LCC Z669.8 .V56 2022 (print) | LCC Z669.8 (ebook) | DDC 020.72/7—dc23/eng/20211028 LC record available at https://lccn.loc.gov/2021028546 LC ebook record available at https://lccn.loc.gov/2021028547 ISBN: 978-1-4408-7593-9 (paperback) 978-1-4408-7594-6 (ebook) 26 25 24 23 22 1 2 3 4 5 This book is also available as an eBook. Libraries Unlimited An Imprint of ABC-CLIO, LLC ABC-CLIO, LLC 147 Castilian Drive Santa Barbara, California 93117 www.abc-clio.com This book is printed on acid-free paper Manufactured in the United States of America Contents Chapter 1. Why Are You Reading This Book? 1 Chapter 2. You Know More Than You Think 7 Chapter 3. H-Indexes, Altmetrics, and Impact Factors—Oh My! 13 Chapter 4. Metrics for All Kinds of Scholars 77 Chapter 5. Open Access: The Good, the Bad, and the Ugly 105 Chapter 6. Finding a Journal That’s the Right Fit 145 Chapter 7. Publish, Don’t Perish! Applying What You’ve Learned 173 Chapter 8. Developing or Finding Metrics Services at Your Library 185 Chapter 9. Avoid Drowning in a Sea of Scholarly Metrics: How to Stay Current 205 Index 213 1 Why Are You Reading This Book? Goodhart’s law “When a measure becomes a target, it ceases to be a good measure.” —Marilyn Strathern This chapter is asking all the stakeholders with an interest in scholarly met- rics (librarians, administrators and department chairs, and researchers) to stop and consider why they are interested in this topic and what they hope to gain from understanding journal ranking and other scholarly metrics. These are complex, but imperfect measurements. When used appropriately, metrics can help users make informed decisions or describe work accom- plishments. When metrics are used to answer questions they were never designed to answer they become meaningless, or worse become an onerous burden. LIBRARIANS Don’t push your agenda. Provide a service; don’t create problems. When we initially investigated how libraries might provide research evaluation services, we noticed two groups emerging from the literature. One group believed libraries should assist with bibliometric services because it was a natural expansion of library services, while the other group focused on the strategic benefits of providing such services. They argued that if libraries provide scholarly metrics services, they would dem- onstrate their worth and reestablish their value to administrators who increasingly rely on metrics. However, coupled with the voices warning 2 DEMYSTIFYING SCHOLARLY METRICS that libraries risked becoming irrelevant if they didn’t evolve to incorpo- rate bibliometric analysis services was a second group warning that the strategic plan to stay relevant might backfire if faculty associate librarians with evaluation reports and failed promotions. Those expressing wariness didn’t oppose libraries offering research evaluation services; instead, the message was to proceed with caution and make sure that faculty would welcome these services. Since our initial investigation, the literature has shifted from “Should libraries provide scholarly metrics services, and if so why?” to “Which services do we provide and how?” The Association of College and Research Libraries (ACRL) updated their information literacy competency standards to a framework model that includes “Scholarship as Conversa- tion” as one the frames, thus recognizing the cycle of research that goes beyond investigation (research) and creation (writing) to sharing (publishing) and communicating (citing, responding, etc.) (Brantley et al., 2017, p. 140). The scholarly communication framework illustrates the ongoing cycle of research, and librarians have recognized needs at each stage of this cycle that they previously focused on—specifically the stages of publishing and disseminating research, which have increasingly grown complex (Si et al., 2019; Ye, 2019). The need to navigate the expanding world of research metrics has created an opportunity for librar- ians to expand their services to include tracking impact and interpreting metrics (Brantley et al., 2017; Howie & Kara, 2020; Powell & Elder, 2019). The need is great enough that many libraries are hiring librarians specifically to help faculty navigate the realm of measuring scholarly out- put (Powell & Elder, 2019). As libraries increase their involvement in the stages of research evaluation and analyzing impact, they have the chance to chase the hottest trends or create faculty center services (Brantley et al., 2017). As more libraries move to incorporate scholarly metric services, past cautions may seem irrelevant. Yet, we believe vigilance is still called for as libraries expand research support to include research evaluation and met- ric analysis services because these services are closely related to profes- sional development and promotions. Librarians must assess whether the programs meet needs or create problems. For example, if librarians attempt to push mandates for faculty to publish in gold open access (OA) journals that either limit their publication options or require high article processing charges (APCs) faculty can’t afford, they’ve created another hurdle to publishing instead of making the process easier. But if faculty are interested in learning more about OA or already decided to prioritize OA journals, then a service that helps faculty publish in OA journals is providing a service that removes hurdles (see chapter 8 for tips on identi- fying faculty needs). Meet a need; don’t create a problem. WHY ARE YOu READING THIS BOOk? 3 ADMINISTRATORS AND DEPARTMENTAL CHAIRS Don’t misapply metrics and create a hostile research environment. Are you hoping a move toward emphasizing bibliometrics will make eval- uating individuals, groups, and the whole institution easier? Are looking for the metrics that will help you evaluate your institution’s scholarly strength in comparison to peer institutions? Are you looking to standardize scholarly metrics to help you compare departments? Are you looking to use metrics to identify research with the most impact potential so your institution allocates funding efficiently? Consider the research environment your decisions will create. Pay close attention to the initial purpose of the metric and what it’s intended to mea- sure before you start using it to measure something else entirely. Understand metrics within their context to avoid unfair comparisons. By ignoring the limitations of metrics, you might pick academic winners and losers by valu- ing subjects that are easier to measure. Remember college applications are a compilation of SAT scores, essays, references, and supplemental information in an attempt to create a complete picture. Additionally, there is a reason job applicants submit a cover letter, resume, and application and run the gaunt- let of interviews: because employers don’t see the full picture from one met- ric. Likewise, author-level metrics and research output reports can’t tell the whole story. We learned that some larger research universities place a high emphasis on improving their institution ranking through scholarly output. These universities make appointment decisions based on evaluating the tal- ent of candidates by producing reports that compare the papers of candi- dates with scholars in the same discipline; these analysis reports are forwarded to the human resources office (Ye, 2019). Metrics are helpful, but they are a piece of the puzzle, not the whole picture. Avoid policies and requirements that shift incentives in research behavior. If metrics are used to promote or elevate areas of study, faculty may lose their freedom to organically study topics of interest and that are important to their field in favor of producing work for the assessment environment (MacColl, 2010). A move to emphasize metrics could create an environment that unfairly punishes fields that produce monographs. Policies implemented to measure scholarly impact may end up stifling scholarly communication if publishing in a journal with a high-impact factor is valued over publishing in a journal that will reach appropriate audiences (MacColl, 2010). Lastly, how you prioritize metrics could result in unreasonable output expectations that pressure researchers to publish more and focus efforts on quantity, not quality, in order to meet unattainable goals. Without carefully considering how your decision may affect the research environment, the requirements (initially established to identify and reward scholarly communication) can become tools in a system that monitors, assesses, and judges researchers. A
The list of books you might like
