Evaluating Research in Academic Journals: A Practical Guide to Realistic Evaluation

169 Pages·2014·3.14 MB·English

by  Fred Pyrczak

Evaluating Research in
Academic Journals
A Practical Guide to
Realistic Evaluation
SIXTH EDITION
Fred Pyrczak
First published 1999 by Pyrczak Publishing.
Published 2017 by Routledge
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
711 Third Avenue, New York, NY 10017, USA
Routledge is an imprint of th e Taylor & Francis Group, an informa business
Copyright © 2014, 2013, 2008, 2005, 2003, 1999 by Taylor & Francis
This edition was written in collaboration with Randall R. Bruce.
All rights reserved. Permission is hereby granted for professors and their students to
photocopy Appendix D for use only in classes in which this book is a required textbook.
Except for Appendix D, no part of this book may be reprinted or reproduced or utilised in
any form or by any electronic, mechanical, or other means, now known or hereafter
invented, including photocopying and recording, or in any information storage or retrieval
system, without permission in writing from the publishers.
Notice:
Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Although the author and publisher have made every effort to ensure the accuracy and
completeness of information contained in this book, we assume no responsibility for
errors, inaccuracies, omissions, or any inconsistency herein. Any slights of people, places,
or organizations are unintentional.
Cover design by Ruben Hornillo.
ISBN-13: 978-1-936-52334-4 (pbk)
ii
Contents
Introduction to the Sixth Edition v
1. Background for Evaluating Research Reports 1
2. Evaluating Titles 13
3. Evaluating Abstracts 23
4. Evaluating Introductions and Literature Reviews 33
5. A Closer Look at Evaluating Literature Reviews 47
6. Evaluating Samples When Researchers Generalize 55
7. Evaluating Samples When Researchers Do Not Generalize 69
8. Evaluating Measures 77
9. Evaluating Experimental Procedures 91
10. Evaluating Analysis and Results Sections: Quantitative Research 103
11. Evaluating Analysis and Results Sections: Qualitative Research 111
12. Evaluating Discussion Sections 123
13. Putting It All Together 131
Appendix A: Quantitative and Qualitative Research: An Overview 137
Appendix B: Examining the Validity Structure of Qualitative Research 141
Appendix C: The Limitations of Significance Testing 149
Appendix D: Checklist of Evaluation Questions 153
Notes

iv
Introduction to the Sixth Edition
When students in the social and behavioral sciences take advanced courses in their
major field of study, they are often required to read and evaluate original research reports
published as articles in academic journals. This book is designed as a guide for students
who are first learning how to engage in this process.
Major Assumptions
First, it is assumed that the students using this book have limited knowledge of
research methods, even though they may have taken a course in introductory research
methods (or may be using this book while taking such a course). Because of this assump-
tion, technical terms and jargon such as true experiment are defined when they are first
used in this book.
Second, it is assumed that students have only a limited grasp of elementary statis-
tics. Thus, the chapters on evaluating statistical reporting in research reports are confined
to criteria that such students can easily comprehend.
Finally, and perhaps most important, it is assumed that students with limited back-
grounds in research methods and statistics can produce adequate evaluations of research
reports—evaluations that get to the heart of important issues and allow students to draw
sound conclusions from published research.
This Book Is Not Written for…
This book is not written for journal editors or members of their editorial review
boards. Such professionals usually have had firsthand experience in conducting research
and have taken advanced courses in research methods and statistics. Published evaluation
criteria for use by these professionals are often terse, full of jargon, and composed of
many elements that cannot be fully comprehended without advanced training and experi-
ence. This book is aimed at a completely different audience: students who are just begin-
ning to learn how to evaluate original reports of research published in journals.
Applying the Evaluation Questions in This Book
Chapters 2 through 13 are organized around evaluation questions that may be an-
swered with a simple “yes” or “no,” where a “yes” indicates that students judge a charac-
teristic to be satisfactory. However, for evaluation questions that deal with complex is-
sues, students may also want to rate each one using a scale from 1 to 5, where 5 is the
highest rating. In addition, N/A (not applicable) may be used when students believe a
characteristic does not apply, and I/I (insufficient information) may be used if the re-
search report does not contain sufficient information for an informed judgment to be
made.
Evaluating Quantitative and Qualitative Research
Quantitative and qualitative research differ in purpose as well as methodology.
Students who are not familiar with the distinctions between the two approaches are ad-
vised to read Appendix A, which presents a very brief overview of the differences, and
Appendix B, which provides an overview of important issues in the evaluation of qualita-
tive research.
Note From the Author
My best wishes are with you as you master the art and science of evaluating re-
search. With the aid of this book, you should find the process both undaunting and fasci-
nating as you seek defensible conclusions regarding research on topics that interest you.
Fred Pyrczak
Los Angeles
vi
Chapter 1
Background for Evaluating
Research Reports
The vast majority of research reports are initially published in academic journals. In
these reports, researchers describe how they identified a research problem, made relevant
observations to gather data, and analyzed the data they collected. The reports usually
conclude with a discussion of the results and their implications. This chapter provides an
overview of some general characteristics of such research. Subsequent chapters present
specific questions that should be applied in the evaluation of research reports.
 Guideline 1: Researchers often examine narrowly defined problems.
Comment: While researchers usually are interested in broad problem areas, they very
often examine only narrow aspects of the problems because of limited resources and
the desire to keep the research manageable by limiting its focus. Furthermore, they
often examine problems in such a way that the results can be easily reduced to
statistics, further limiting the breadth of their research.1
Example 1.1.1 briefly describes a study on two correlates of prosocial behavior
(i.e., helping behavior). To make the study of this issue manageable, the researchers
greatly limited its scope. Specifically, they examined only one very narrow type of
prosocial behavior (making donations to homeless men who were begging in public).
Example 1.1.1
A study on prosocial behavior, narrowly defined:
In order to study the relationship between prosocial behavior and gender as well
as age, researchers located five men who appeared to be homeless and were
soliciting money on street corners using cardboard signs. Without approaching
the men, the researchers observed them from a short distance for two hours
each. For each pedestrian who walked within ten feet of the men, the researchers
recorded whether the pedestrian made a donation. The researchers also recorded
the gender and approximate age of each pedestrian.
Because researchers often conduct their research on narrowly defined problems,
an important task in the evaluation of research is to judge whether a researcher has
1 Qualitative researchers (see Appendices A and B) generally take a broader view when defining a problem
to be explored in research and are not constrained by the need to reduce the results to numbers and
statistics.
1
Chapter 1 Background for Evaluating Research Reports
defined the problem so narrowly that it fails to make an important contribution to the
advancement of knowledge.
 Guideline 2: Researchers often conduct studies in artificial settings.
Comment: Laboratories on university campuses are often the settings for research. To
study the effects of alcohol consumption on driving behavior, a group of participants
might be asked to drink carefully measured amounts of alcohol in a laboratory and
then “drive” using virtual-reality simulators. Example 1.2.1 describes the preparation
of the cocktails in a study of this type.
Example 1.2.12
Alcoholic beverages prepared for consumption in a laboratory setting:
The preparation of the cocktail was done in a separate area out of view of the
participant. All cocktails were a 16-oz mixture of orange juice, cranberry juice,
grapefruit juice (ratio 4:2:1, respectively). For the cocktails containing alcohol,
we added 2 oz of 190-proof grain alcohol mixed thoroughly. For the placebo
cocktail, we lightly sprayed the surface of the juice cocktail with alcohol using
an atomizer placed slightly above the juice surface to impart an aroma of alcohol
to the glass and beverage surface. This placebo cocktail was then immediately
given to the participant to consume. This procedure results in the same alcohol
aroma being imparted to the placebo cocktail as the alcohol cocktail.…
Such a study might have limited generalizability to drinking in out-of-laboratory
settings, such as nightclubs, the home, picnics, and other places where those who are
consuming alcohol may be drinking different amounts at different rates while consum-
ing (or not consuming) various foods. Nevertheless, conducting such research in a
laboratory allows researchers to simplify and control variables such as the amount of
alcohol consumed, the types of food being consumed, the number of individuals in a
car, and so on. In short, researchers very often opt against studying variables in com-
plex, real-life settings for themore interpretable research results typically obtained in a
laboratory.
 Guideline 3: Researchers use less-than-perfect methods of observation.
Comment: In research, observation can take many forms—from paper-and-pencil
multiple-choice achievement tests to essay examinations, from administering a
paper-and-pencil attitude scale with choices from “strongly agree” to “strongly dis-
agree” to conducting unstructured interviews to identify interviewees’ attitudes.3 Of
2 Barkley, R. A., Murphy, K. R., O’Connell, T., Anderson, D., & Connor, D. F. (2006). Effects of two dos-
es of alcohol on simulator driving performance in adults with attention-deficit/hyperactivity disorder. Neu-
ropsychology,20, 77–87.
3 Researchers sometimes refer to measurement tools for making observations as instruments, especially in
older literature.
2
Chapter 1 Background for Evaluating Research Reports
course, observation also includes direct observation of individuals interacting in either
their natural environments or laboratory settings.
It is safe to assume that all methods of observation are flawed to some extent. To
see why this is so, consider a professor/researcher who is interested in studying racial
relations in society in general. Because of limited resources, the researcher decides to
make direct observations of White and African American students interacting (and/or
not interacting) in the college cafeteria. The observations will necessarily be limited to
the types of behaviors typically exhibited in cafeteria settings—a weakness in the re-
searcher’s method of observation. In addition, observations will be limited to certain
overt behaviors because, for instance, it will be difficult for the researcher to hear most
of what is being said without obtruding on the privacy of the students.
On the other hand, suppose that another researcher decides to measure racial at-
titudes by having students respond anonymously to racial statements by circling
“agree” or “disagree” for each one. This researcher has an entirely different set of
weaknesses in the observational method. First is the matter of whether students will
reveal their real attitudes on such a scale—even if the response is anonymous—
because most college students are aware that negative racial attitudes are severely
frowned on in academic communities. Thus, some students might indicate what they
believe to be socially desirable (i.e., socially “correct”) rather than reveal their true at-
titudes.
In short, there is no perfect way to measure complex variables. Instead of ex-
pecting perfection, a consumer of research should consider this question: Is the method
sufficiently valid and reliable to provide potentially useful information?
Examples 1.3.1 and 1.3.2 show statements from research articles in which the
researchers acknowledge limitations in their observational methods.
Example 1.3.14
Researchers’ acknowledgment of alimitation of their measures:
In addition, the assessment of marital religious discord was limited to one item.
Future research should include a multiple-items scale of marital religious dis-
cord and additional types of measures, such as interviews or observational cod-
ing, as well as multiple informants.
Example 1.3.25
Researchers’ acknowledgment of limitations of self-reports for observation:
Despite these strengths, this study is not without limitations. First, the small
sample size decreases…. Fourth, neighborhood danger was measured from
mothers’ self-reports of the events which occurred in the neighborhood during
4 Kor, A., Mikulincer, M., & Pirutinsky, S. (2012). Family functioning among returnees to Orthodox Juda-
ism in Israel.Journal of Family Psychology,26, 149–158.
5 Callahan, K. L., Scaramella, L. V., Laird, R. D., & Sohr-Preston, S. L. (2011). Neighborhood disadvan-
tage as a moderator of the association between harsh parenting and toddler-aged children’s internalizing
and externalizing problems.Journal of Family Psychology,25, 68–76.
3