Editors' Afterword

13 Pages·2016·0.27 MB·English

Special Issue
InternationalJournalofQualitativeMethods
January-December2016:1–13
Editors’ Afterword: Toward Evidence-Based ªTheAuthor(s)2016
Reprintsandpermissions:
Guidelines for Reviewing Mixed Methods sagepub.com/journalsPermissions.nav
DOI:10.1177/1609406916628986
Research Manuscripts Submitted to Journals ijqm.sagepub.com
Anthony J. Onwuegbuzie1 and Cheryl Poth2
Reviewing Mixed Methods Research Manuscripts withthemissionofthejournal,(g)treatauthorsethically,and
(h) maintain standards of excellence for the journal and the
Our work as guest coeditors for this special issue was greatly
field/discipline (Fischer, 2011; Glogoff, 1988). In essence,
assistedbyourreferees.Inthisissue,wepresentedsixofmore
then, the journal reviewers, almost always numbering more
than1millionscholarlyjournalarticlesthatwerepublishedthis
thanoneforeachsubmittedmanuscript,togetherserveasedi-
year.Indeed,according toWare (2006),
torial gatekeepers and, in turn, gatekeepers of new academic
knowledge, wherein gatekeeping suggests that those manu-
Thereareapproximately23,000scholarlyjournalsintheworld, scripts‘‘thatmakeauniquecontributiontotheexistinglitera-
collectivelypublishing1.4millionarticlesayear.Thenumber turethroughnoveltyofideas,cleardevelopmentoftheory,and
ofarticlespublishedeachyearandthenumberofjournalshave
rigor of methods receive higher ratings and more favorable
both grown steadily for over two centuries, by about 3% and
editorial decisions’’ (Gilliland &Cortina, 1997,p.428).
3.5%peryear,respectively.Thereasonistheequallypersistent
As noted by Fischer (2011), in general, reviewers should
growthinthenumberofresearchers,whichhasalsogrownat
provide reviews that are constructive, comprehensive, thor-
about3%peryearandnowstandsataround5.5million.(p.3)
ough, balanced, diplomatic, open minded, respectful of the
author’s ideas and property rights, and prompt. Even more
ConsistentwithWare(2006),Bjo¨rk,Roos,andLauri(2009)
importantly, reviewers always shouldstrive tomaximize non-
estimatedatotalof23,750journalsin2006,publishingatotal
maleficence(i.e.,notcausingharmtotheauthors);beneficence
of 1,346,000 articles during the year. Moreover, currently,
(i.e., working for the benefit of authors); (social) justice (i.e.,
there are more than 50 million published scholarly journal
providingreviewsbasedonuniversalprinciplesandrules,inan
articles in existence (Jinha, 2010). With this number of scho-
impartialandwarrantedmannerinordertoguarantee fairand
larlyarticles,itisclearthat‘‘journalpublicationistheprimary
equitabletreatmentofallauthors);fidelity(i.e.,demonstrating
methodtodisseminatescholarlyinformationandresearchfind-
faithfulness, loyalty, and commitment to the journal); profes-
ings’’ (Glogoff, 1988, p. 400)—thereby being the foremost
sional competence (i.e., writing reviews that are within the
pathwaytonewcontributionstoacademic-related knowledge.
reviewer’s set of skills and knowledge of the topic explored,
Significant to the formal communication process are the jour-
methodology and methods used, and the results reported);
nal reviewers (i.e., referees) who, without enumeration, per-
integrity (i.e., being fair, honest, and respectful of author’s
form at least the following functions: (a) assess the quality of
ideas, procedures, and findings); scholarly responsibility (i.e.,
a manuscript and the importance of the underlying topic,
adhering to best practices through reviews that are warranted
(b) assess the priority or innovation of the submitted manu-
script within the context of the extant literature, (c) provide a
review that helps the editor make an informed decision about
1DepartmentofEducationalLeadershipandCounseling,SamHoustonState
the disposition of the submitted manuscript (e.g., accept vs.
University,Huntsville,TX,USA
rejectvs.reviseandresubmit),(d)provideinformedguidance 2Faculty of Education, Department of Educational Psychology, University of
totheeditorandtheauthor(s)regardingmodificationsthatwill Alberta,Edmonton,Canada
help improve the submitted manuscript in a manner that will
CorrespondingAuthor:
increase its potential to contribute to the literature, (e) relieve
Anthony J. Onwuegbuzie, Department of Educational Leadership and
the pressure of an unfavorable decision from the editor,
Counseling,Box2119,SamHoustonStateUniversity,Huntsville,Texas.
(f) facilitate the dissemination of knowledge that is consistent Email:tonyonwuegbuzie@aol
CreativeCommonsCC-BY-NC:ThisarticleisdistributedunderthetermsoftheCreativeCommonsAttribution-NonCommercial3.0License
(http://www.creativecommons.org/licenses/by-nc/3.0/)whichpermitsnon-commercialuse,reproductionanddistributionoftheworkwithoutfurther
permissionprovidedtheoriginalworkisattributedasspecifiedontheSAGEandOpenAccesspage(https://us.sagepub.com/en-us/nam/open-access-at-sage).
2 International Journal of Qualitative Methods
and transparent and that involve self-reflection and meta- appraisal of manuscripts of our special issue reviewers to
cognition on the part of the reviewer); social responsibility develop a framework for comprehensively reviewing mixed
(i.e., demonstrating awareness of the social dimensions of the methodsresearch manuscripts.
author’s topic); and respecting rights, dignity, and diversity
(i.e., striving to eliminate bias for misrepresenting authors’
worksandnotdiscriminatingauthorsbasedonexceptionalities Method
oftheirtopics;cf.Onwuegbuzie&Frels,2016)—which,ifall
Research Design
elements are followed, would yield a reviewer who is meta-
ethical, which implies that he or she adheres to virtue ethics Our study involved combining quantitative and qualitative
(i.e.,referringtothecharacterofthereviewer—asopposedto approaches within a case study—yielding what Onwuegbuzie
the standards of excellence for the journal—providing the (2015) referred to as a mixed methods case study (MMCS)—
impetus for ethical reviews) and pragmatic ethics (i.e., using specifically, a qualitative-dominant MMCS. The case was
thestandardssetbythejournaleditorundertheassumptionthat boundedbythemanuscriptssubmittedtothetwoInternational
theeditorisprogressingmorallyinlinewiththeprogressionof Journal of Qualitative Methods (IJQM) mixed methods
scientificknowledge). research special issues. These two special issues yielded
Over the years, a few researchers have examined the 20manuscriptsthatweselectedasguestcoeditors(i.e.,criterion
factors that determine reviewers’ appraisal of manuscripts sampling;Onwuegbuzie&Collins,2007)fromthe70proposals
submitted to journals for review for possible publication. submitted,basedonthequalityandfit(i.e.,it[potentially]pro-
For example, in Gilliland and Cortina’s (1997) study, vided a compelling example of how mixed methods research
116 reviewers from five rehabilitation counseling-related informsandenhancesqualitativeresearch—consistentwithour
journals completed a 63-item questionnaire design to extract call for the special issue). From the 20 sets of authors whose
information on common reasons for manuscript acceptance proposals were accepted, 16 submitted full manuscripts within
or rejection. These reviewers identified 13 manuscript char- the5-monthdeadlineforsubmission.
acteristics that were deemed to be important determinants of Afterreceivingthesubmissions,wesoughtthreereviewsfor
reviewer acceptance or rejection, which represented either eachmanuscriptaspartofaquadruple-blindpeer-reviewpro-
negative influences or positive influences. Respectively, the cess (i.e., during the initial review process, [a] the peer
eight negative influences were direct replications that added reviewers were not aware of author identification, [b] each
little to theory development (51%); topics that present mate- author(s)wasnotawareoftheidentityofthereviewersofthe
rial well outside the mainstream of the field (47%); manu- manuscript,[c]theactioneditorwasnotawareofauthoriden-
scripts that represent pilot studies with little evidence of tificationatthetimethattheeditorial decisionwasmade,and
generalizability (43%); manuscripts that are too lengthy [d] the action editor was not aware of the reviewers of the
(36%); application of inappropriate analysis (e.g., para- manuscript at the time that the editorial decision was
metric test for ordinal data; 35%); manuscripts containing made)—with the exception of the final manuscript that was
only secondary analysis of data presented by others (32%); reviewedforwhichtheactioneditorwasawareoftheidentity
experimental data with no control group (29%); and studies oftheauthor(s),whichyieldedatriple-blindreview.Itisnote-
lacking statistical significance, whether they are based on worthy that for each manuscript, we selected at least one
either new or currently popular theories (28%). Contrast- reviewer who was a recognized mixed methods research
ingly, respectively, the five positive influences were manu- expert.Together,thereviewers providedtheauthorsfeedback
scripts representing a new, original theory (33%); author thatwas greater thanthe sum ofthe individualreviews.
analyzing interval data appropriately (29%); manuscript Across the 16 submitted manuscripts, 45 reviews of the
with content of interest to the field but differing in content initialsubmissionswereobtained,whichbecamethefinalsam-
from those traditionally published in the journal (23%); new pleforourMMCSstudy.Eachofthese45individualreviewers
statistical methods, including data collection techniques provided her or his own feedback based on six criteria (i.e.,
(20%); and author’s reputation (20%). relevance,qualityofinformation,qualityofwriting,conform-
However, all the studies examining factors that determine ingtoAmerican Psychological Association[APA] guidelines,
reviewers’ appraisal of manuscripts submitted to journals for adherence to ethical standards, and suggestions for improve-
reviewfor possible publication have involved the appraisal of ment).Oneoftheeditorsthencompiledthefeedbacktogether
monomethod manuscripts that represented the quantitative and pointed out the general themes to the author while also
research tradition only or the qualitative research tradition giving the authors access to the individual reviews. We were
only. Indeed, to date, no studies exist examining factors that unusuallyfortunateaseditorsthatallourreviewerswererecog-
determine reviewers’ appraisal of manuscripts submitted to nizedexpertsintheirfields,timelyintheircontributionstothis
journals that represent the mixed methods research tradition. process,andextraordinarilyhelpfulintheirencouragementand
Therefore, for this editorial, we decided to investigate these critiques. Indeed, several of our authors commented on the
factors. In particular, our primary goal was to begin the con- mentoring that they received via the reviewers’ feedback, and
versationamongmembersofthemixedmethodsresearchcom- we are grateful to our reviewers for providing the learning
munity regarding what makes a quality review by using the experience for allofus.
Onwuegbuzie and Poth 3
Table1.FrequencyandPrevalenceRatesPertainingtotheMeta-ThemesExtractedFromReviewers’CommentstoManuscriptsSubmittedfor
theTwoInternationalJournalofQualitativeMethodsSpecialIssuesinMixedMethodsResearch.
MixedMethodsReviewers NonmixedMethodReviewers TotalReviewers
StrengthsVersus.Limitations Meta-Theme (n¼17) (n¼28) (N¼45)
Limitations Lackofwarrantedness 13(76.5%) 17(60.7%) 30(66.7%)
Lackofjustification 10(58.8%) 20(71.4%) 30(66.7%)
Writingissues 10(58.8%) 15(53.6%) 25(55.6%)
Lackoftransparency 7(41.2%) 8(28.6%) 15(33.3%)
Lackofintegration 5(29.4%) 5(17.9%) 10(22.2%)
Philosophicalissues 2(11.8%) 2(7.1%) 4(8.9%)
Strengths Positiveinfluences 7(41.2) 15(53.6) 22(48.9%)
Analysis both editions of the Handbook of Mixed Methods Research.
In addition to a descriptive analysis, for the mixed methods
Weconductedasequentialmixedmethodsanalysis(Onwueg-
reviewers, we conducted a correspondence analysis to factor
buzie & Teddlie, 2003) to analyze the data. This analysis
the meta-themes with their associations in at least a two-
involved us conducting a qualitative analysis followed by a
dimensional map (Onwuegbuzie, Leech, Dickinson, & Zoran,
quantitative analysis. Specifically, we used constant compari-
2010). A correspondence analysis is a multivariate and visual
son analysis (Glaser, 1965) to analyze qualitatively all of the
technique for conducting a quantitative analysis of emergent
reviewers’comments.Thisanalysisinvolvedcodingchunksof
themesormeta-themes(Michailidis,2007).Onwuegbuzieand
words,byeachreviewer,intomeaningfulunitsofwords,phrases,
Combs (2010) referred to a correspondence analysis as repre-
sentences,orparagraphsthatdescribedthecontentsoftheseg-
senting a crossover mixed analysis, whereby we usedthe anal-
mented data. These codes represented the underlying themes
ysis types associated with one tradition (i.e., quantitative
withineachmeta-theme,whichwereidentifiedaposteriori(Con-
analysis: descriptive analysis and correspondence analysis) to
stas,1992).Ourcodesandlocusoftypology(i.e.,theme)devel-
analyzedataassociatedwithadifferenttradition(i.e.,qualitative
opment were investigative, arising from our own constructions
data:emergentmeta-themes).WeusedQDAMiner4.1.23(Pro-
(Constas,1992).Afterthecodingprocess,wetransformedboth
valisResearch,2015)toconductourcorrespondenceanalysis.
the meta-themes and themes into numerical data that could be
analyzedqualitatively—aprocessknownasquantitizing(Miles
Results
& Huberman, 1994; Onwuegbuzie & Teddlie, 2003; Sande-
lowski,Voils,&Knafl,2009;Tashakkori&Teddlie,1998).
Ideally, consistent with our qualitative dominant MMCS
Oncewe hadquantitized thedata,we conductedadescrip-
design, in our results section, we had intended to present an
tiveanalysisofthequantitizedmeta-themesandthemes,which
array of rich quotations directly from the voices of the
involved determining the frequency (i.e., by counting the
reviewers.However,duetoourveryshorttimeframeforwrit-
number of reviewers who contributed to each theme and
ingoureditorial,wedidnothavetimetoobtainthepermission
meta-theme)andprevalencerates(i.e.,bydeterminingthepro-
ofall45reviewersaswellasfromallthe16setsofauthorsto
portionofreviewerswhocontributedtoeachthemeandmeta-
include these quotations. Also, each quotation that we would
theme),forthefullsetofreviewers(i.e.,n¼45)aswellasfor have provided likely would have revealed the identity of the
thereviewerswhohadmixedmethodsresearchexpertise(i.e.,
manuscripttowhichwewerereferringto.Andwedidnotwant
mixedmethodsreviewers;n¼17)andthereviewerswhodid togiveanyimpressionthatweweresinglingoutanyparticular
nothavemixedmethodsresearchexpertisebutwhohadexper-
set of authors for criticism. Thus, in providing the following
tiseinotherareassuchasinquantitativeresearchorqualitative
findings, we have omitted the quotations that were present in
research(i.e.,nonmixedmethodsreviewers;n¼28).Notably, earlierdraftsofthiseditorial.However,afterwehaveobtained
all our mixed methods reviewers were leading mixed metho-
permission from the reviewers and authors, we do intend to
dologistsinthefieldwith(a)onaverage,thesereviewershav-
present these quotations in a follow-up work such that
ing more than 16 years of academic experience (i.e., M ¼ reviewers can hear the voices of the reviewers and have a
16.47, SD ¼ 12.24); (b) slightly more than one half of them sampleofthequalityofthereviewsthatwereceived,forwhich
(52.9%) beginning their careers as mixed methodologists
we areso appreciative.
beforethepublicationofthelandmarkfirsteditionoftheHand-
book of Mixed Methods Research (Tashakkori & Teddlie,
Meta-Themes That Represented a Limitation
2003); (c) nearly one third of them (i.e., 29.4%) being
authors/coauthors of mixed methods research textbooks; The constant comparison analysis yielded the following six
(d) nearly one half of them (i.e., 47.1%) being authors/coau- meta-themesthateachrepresentedalimitationofoneormore
thors of methodology (i.e., quantitative, qualitative, and/or ofthe16manuscripts:lackofwarrantedness,lackofjustifica-
mixed research) textbooks; and (e) nearly one half of them tion, writing issues, lack of transparency, lack of integration,
(i.e., 47.1%) being authors/coauthors of a chapter in one or and philosophical issues. Each of these meta-themes will be
4 International Journal of Qualitative Methods
Both Empirical and
Conceptual/Theoretical
Lack of Integration
/Methodological
Manuscripts
M E M
M
E Lack of Justification
E
E
M
M M Both Empirical and
E Conceptual/Theoretical
/Methodological
M M M
Writing Issues Manuscripts
Philosophical
Issues Lack of All Empirical
Warrantedness Manuscripts
E
All E E
Conceptual/Theoretical
/Methodological
Lack of
Manuscripts
Transparency
Figure1.Correspondenceanalysisplotofthesixemergentmeta-themesassociatedwithlimitationsasafunctionofgenreofmanuscript.
E¼empiricalmanuscript;M¼conceptual/theoretical/methodologicalmanuscript.
described in subsequent sections. Table 1 displays the fre- empirical manuscripts. Finally, the lack of integration, lack
quency and prevalence rates pertaining to the meta-themes of justification, and writing issues meta-themes were repre-
extracted from the reviewers’ comments to the 45 submitted sented by both conceptual/theoretical/methodological manu-
manuscripts for all reviewers and as a function of type of scripts andempirical manuscripts.
reviewer. It can be seen from this table that for the mixed
methods reviewers, lack of warrantedness was the most pre- Meta-theme 1: Lack of warrantedness. This meta-theme, repre-
valentmeta-theme,withslightlymorethanthreequarters(i.e., senting the most prevalent meta-theme, echoes the reporting
76.5%) of these reviewers being classified under this meta- criteriaasdescribedintheseminaldocumentdevelopedbythe
theme. That is, mixed methods reviewers were more likely to Task Force on Reporting of Research Methods in American
provide a criticism of a manuscript that was classified as rep- Educational Research Association (AERA) Publications and
resentinglackofwarrantednessthananyothermeta-theme.In adoptedbytheAERACouncilin2006,whereinitisstipulated
contrast, for the nonmixed methods reviewers, lack of justifi- that reports of empirical research should be warranted inas-
cationwasthemostprevalentmeta-theme,withmorethantwo much as adequate evidence should be provided to justify the
thirds (i.e., 71.4%) of these reviewers being classified under resultsandconclusions(AmericanEducationalResearchAsso-
thismeta-theme. ciation[AERA],2006).Thus,thelackofwarrantednessmeta-
Figure 1 illustrates the reviews of the 17 mixed methods theme refers to any criticism made by a reviewer regarding
reviewersmapped,viacorrespondenceanalysis,ontothespace inappropriate, inadequate, or missing evidence. This meta-
thatdisplaysthesixemergentmeta-themes.Thisfigureshows theme contained the following seven themes: (a) lack of evi-
how the reviews of the mixed methods reviewers related to dence, (b) insufficient findings, (c) old/inadequate sources
each other in regard to these six meta-themes. In particular, used, (d) lack ofdefinition, (e) no/inadequate reference tothe
the lack of warrantedness and philosophical issues meta- most recent mixed methods research literature, (f) reference
themes exclusively were represented by conceptual, theoreti- listerrors,and(g)citationerrors.Thelackofevidencetheme
cal, or methodological manuscripts. In contrast, the lack of referstotheauthormakingastatementanywhereinthemanu-
transparency meta-theme exclusively was represented by scriptthatwasclearlybiased,overlyjudgmental,nonscholarly,
Onwuegbuzie and Poth 5
or the like, or, even if the statement was reasonable, lacked Table2.Meta-Theme1:LackofWarrantedness:ThemesExtracted
citations.Theinsufficientfindingstheme,asthetermsuggests, FromReviewers’Comments.
arosewhentheauthoromittedoneormoreimportantfindings
Mixed Nonmixed
that were needed to address one or more research questions
Methods Method Total
and/ortotestoneormorehypotheses,oroneormorefindings Reviewers Reviewers Reviewers
that were presented lack sufficient depth. The old/inadequate Theme (n¼17) (n¼28) (N¼45)
sourcesusedthemeimpliesthatcitationswereprovided;how-
Lackofevidence 7(41.2%) 9(32.1%) 16(35.6%)
ever,asignificantproportionofthemwereeitherdatedorwere
Insufficientfindings 4(23.5%) 7(25.0%) 11(24.4%)
not appropriate for the claim made. The lack of definition
Old/inadequatesourcesused 4(23.5%) 5(17.9%) 9(20.0%)
theme means that one or more terms were introduced without
Lackofdefinition 6(35.3%) 3(10.7%) 9(20.0%)
anydefinitionand/orexplanation.Theno/inadequatereference No/inadequatereferenceto 4(23.5%) 2(7.1%) 6(13.3%)
to the most recent mixed methods research literature theme themostrecentmixed
occurred when the author did not provide the most up-to-date methodsresearch
citation(s)fromthemixedmethodsresearchliteratureatoneor literature
Referencelisterrors 3(17.6%) 2(7.1%) 5(11.1%)
moreofthe12components ofaprimary researchreport iden-
Citationerrors 1(5.9%) 1(3.6%) 2(4.4%)
tified by Onwuegbuzie and Frels (2016): problem statement,
literature review, theoretical/conceptual framework, research
question(s), hypotheses, participants, instruments, procedures,
errors. Thus, citation errors are not only errors that ethically
analyses, interpretation of the findings, directions for future
mustbeavoided,butalsotheirpresenceispredictive ofunsa-
research, and implications for the field. The reference list
tisfactory manuscripts tosomedegree.
errors theme occurred when the author provided one or more
Table 2 displays the frequency and prevalence rates of
referencesinthereferencelistthatcontainedanerrorofomis-
themes pertaining to the lack of warrantedness meta-theme
sionorcommission. And, asnoted byAPA] (2010, p.180),
themesextractedfromthereviewers’commentstothe45sub-
mittedmanuscriptsforallreviewers,andasafunctionoftype
Becauseonepurposeoflistingreferencesistoenablereadersto
of reviewer. It can be seen from this table that the lack of
retrieveandusethesources,referencedatamustbecorrectand
evidencethemewasthemostprevalentforbothmixedmethods
complete .... Authors are responsible for all information in
reviewers andnonmixed methodsreviewers.
theirreferencelists.Accuratelypreparedreferenceshelpestab-
lishyourcredibilityasacarefulresearcher.
Meta-theme 2: Lack of justification. This meta-theme was perti-
Unfortunately, reference list errors are rampant among nentwhentheauthordidnotmakecleartheimportanceofthe
authors, with authors committing more than 12 reference list study in some way. This meta-theme contained the following
errors per manuscript, on average (Onwuegbuzie, Hwang, six themes: (a) Underdeveloped, (b) Lack of significance,
Frels, & Slate, 2011). Yet, manuscripts that contain more ref- (c)Didnotadvancequalitativeresearch,(d)Didnotadvance
erence list errors than this average are statistically and practi- mixedmethodsresearch,(e)Lackofrationale,and(f)Lackof
cally (Cohen’s [1988] d¼ 0.83)significantly less likelytobe purpose statement. The underdeveloped theme refers to a
acceptedforpublicationthanaremanuscriptswithmuchfewer manuscriptcontainingaconceptual,theoretical,ormethodolo-
reference list errors than this average (Onwuegbuzie et al., gicalessaywhereintheunderlyingassumptions,ideas,beliefs,
2011). propositions,theories,schemas,models,hypotheses,orthelike
Finally,thecitationerrorsthemereflectsafailure‘‘tomake havenotbeenexplicatedsufficiently.Or,inthecaseofamixed
certainthateachsourcereferencedappearsinbothplaces[text methodsresearchstudy,thisthemeimpliesthatoneormoreof
and reference list] and that the text citation and reference list theelementsoftheinquiry(e.g.,conceptualframework,theo-
entryareidenticalinspellingofauthornamesandyear’’;APA, retical framework) have not been discussed sufficiently.
2010, p. 174). Disturbingly, citation errors are rampant in Whereassomereviewerspointedtoelementsofthemanuscript
manuscripts of all genres—with as many as 91.8% of authors thatwereunderdeveloped,insomecases,thewholemanuscript
committingoneormorecitationerrors(Onwuegbuzie,Frels,& was described as being underdeveloped, with the reviewer
Slate,2010).Interestingly,Onwuegbuzie,Waytowich,andJiao usingawordlike‘‘superficiality.’’Inmanyinstanceshere,the
(2006)reportedthatmanuscriptssubmittedtotheResearchin reviewer criticized the manuscript for not being true to the
theSchoolsjournalthatcontainmorethanthreecitationerrors special issue theme of enhancing qualitative research through
areapproximately4timesmorelikely(oddsratio¼4.01;95% mixedmethods.Thelackofsignificancethemearosewhenthe
confidence interval ¼ 1.22, 13.17) to be rejected than are author did not make clear the (educational) significance of
manuscripts with three or less citation errors. Consistent with the study or the conceptual/theoretical/methodological essay.
thisfinding,Onwuegbuzie,Frels,etal.(2010)documentedthat The did not advance qualitative research theme was particu-
manuscriptswithmorecitationerrorsarestatisticallyandprac- larlypertinentforthissetofmanuscriptsbecausethethemeof
tically(Cohen’s[1988]d¼0.45)significantlylesslikelytobe the special issue was ‘‘How mixed methods informs and
acceptedforpublicationthanaremanuscriptswithlesscitation enhancesqualitativeresearch.’’Similartothedidnotadvance
6 International Journal of Qualitative Methods
Table3.Meta-Theme2:LackofJustification:ThemesExtractedFrom That is, the rationale, purpose statement, research questions,
Reviewers’Comments. and educational significance, respectively, were not presented
at the end of the introduction section—after the literature
Nonmixed
reviewsection—suchthatintheintroductionsection,theliter-
MixedMethods Method Total
Reviewers Reviewers Reviewers aturereviewidentifiedtheknowledgebase,therationaleiden-
Theme (n¼17) (n¼28%) (N¼45%) tified a significant gap in the literature, and the purpose
statement delineated how the author(s) attempted to fill this
Underdeveloped 7(41.2%) 14(50.0%) 21(46.7%)
gap. Alternatively, the author could have presented the ratio-
Lackofsignificance 2(11.8%) 8(28.6%) 10(22.2%)
nale, purpose statement, and research questions, respectively,
Didnotadvance 4(23.5%) 4(14.3%) 8(17.8%)
early on in the introduction section (e.g., first or second para-
qualitativeresearch
Didnotadvance 5(29.4%) 3(10.7%) 8(17.8%) graph). This would be followed by a section entitled such as
mixedmethods ‘‘ReviewoftheRelatedLiterature,’’whichcontainedtheliter-
research ature review, theoretical framework, and educational signifi-
Lackofrationale 1(5.9%) 5(17.9%) 6(13.3%) cance. In any case, the author’s lack of structure adversely
Lackofpurpose 0(0%) 2(7.1%) 2(4.4%)
affected the flow of the manuscript as well as the ease with
statement
which the logic of the article could be followed. That is, this
lackofstructurepreventedtheintroductionsectionfromflow-
inginamaximalway.Withrespecttothemethodsection,the
qualitativeresearchtheme,theDidnotadvancemixedmethods
author who was criticized did not use the standard partici-
research occurred when the value added for conducting a
pants—instruments—procedure—analysis format.
mixed methods research study over a monomethod research
Therepetitiontheme,asitslabelsuggests,cametothefore
studywas notapparent.
whentheauthorunnecessarilyrepeatedinformationatvarious
Thelackofrationalethemeoccurredwhenauthorsdidnot
pointsofthemanuscript.Theinappropriatewordusagetheme
providearationaleeitherfortheirstudyorfortheirconceptual/
occurred when the author used an inappropriate word to
theoretical/methodological essay. With respect to empirical
describe a concept. One example of this was an author mista-
studies, criticism associated with this theme also arose when
kenly using the words ‘‘methodology’’ and ‘‘method’’ inter-
the author did not provide a rationale for mixing. Finally, the
changeably. Yet, these terms are very different. Whereas a
lackofpurposestatementthemeoccurredwhenauthorsdidnot
methodology can be defined as a broad approach to scientific
specifythepurposeeitheroftheirstudyoroftheirconceptual/
inquiry with general preferences for certain types of designs,
theoretical/methodological essay. Table 3 displays the fre-
sampling logic, analytical strategies, and so forth, methods
quency and prevalence rates of themes pertaining to the lack
include specific strategies andprocedures for research design,
ofjustificationmeta-themeextractedfromthereviewers’com-
sampling, datacollection, analysis, and thelike.
mentstothe45submittedmanuscriptsforallreviewers,andas
Finally, the lack of transitions theme occurred when the
afunctionoftypeofreviewer.Itcanbeseenfromthistablethat
authordidnotadequatelylinksentences.Inparticular,inthese
theundeveloped themewas themost prevalent.
instances, the author did not use link words to make these
connections. Yet, link words/phrases are very useful for con-
Meta-theme3:Writingissues.Thismetatheme,whichpertainsto necting ideas and, hence, for connecting sentences and para-
thequalityandeffectivenessoftheauthors’writing,contained graphs. Interestingly, analyzing manuscripts that were
the following six themes: (a) Lack of clarity, (b) Lack of con- submittedovera3-yearperiod(i.e.,2011–2014)tothejournal
sistency,(c)Lackofstructure,(d)Repetition,(e)Inappropriate Research in the Schools (RITS), for which one of the special
wordusage,and(f)Lackoftransitions.Specifically,comments issueeditorsservesasaneditor,viatheuseofQDAMiner4.1
relating tolack ofclarity occurred whenone ormore sections and WordStat 6.0, Onwuegbuzie (2016) documented that the
of the manuscript was not clearly written. Also adversely dimension labeled as add information/provide similarity was
impactingtheclarityofthemanuscriptwaslackofconsistency, the most commonly used (by 71.6% of authors), followed by
which, interestingly, was only pointed out by the nonmixed thedimensionlabelednarration(by60.8%ofauthors),andthe
methodsreviewers(i.e.,n¼5).Thisthemewasapparentwhen dimension labeled sequence previous ideas (also 60.8% fre-
thereweretwoormorestatementsthatcontradictedeachother. quency). The remaining nine dimensions were used by less
For instance, one author presented information in the abstract than 50% of the authors. The three most common link words,
that was not consistent with information provided in the body respectively,werefinally(52.7%),similarly(52.7%),andaddi-
ofthemanuscript. Interms ofthe lackofstructure theme,the tionally (51.4%). All other link words/phrases were used by
reviewer making this observation tended to criticize the one third of the authors or less. Even more compelling was
author’s introduction section and/or method section. With Onwuegbuzie and Frels’s (2016) finding that the following
regardtotheintroductionsection,theauthorwhowascriticized dimensions statistically significantly and practically signifi-
did not use the following standard format: literature review— cantly predicted whether or not a manuscript was rejected by
theoretical framework—rationale—purpose statement— the editor: add information/provide similarity, narration, and
research questions—hypothesis—educational significance. provide an emphasis. More specifically, manuscripts that
Onwuegbuzie and Poth 7
Table 4. Meta-Theme 3: Writing Issues: Themes Extracted From average of 12.6 in pharmacology/toxicology to an average of
Reviewers’Comments. 25.6 in mathematics, all fell into the very difficult range or
college grade level (i.e., 1–30), the highest level of text
MixedMethods NonmixedMethod Total
difficulty.
Reviewers Reviewers Reviewers
Theme (n¼17) (n¼28) (N¼45) Metoyer-Duran (1993) examined whether readability esti-
mates differed significantly among published, accepted, and
Lackofclarity 9(52.9%) 7(25.0%) 16(35.6%) rejectedmanuscriptsandabstractsfromCollegeandResearch
Lackofconsistency 0(0%) 5(17.9%) 5(11.1%)
Librariesduringthe1990–1991period.Thisresearcherascer-
Lackofstructure 0(0%) 2(7.1%) 2(4.4%)
tained that the readability estimates of manuscripts accepted
Repetition 1(5.9%) 1(3.6%) 2(4.4%)
forpublicationweresignificantlydifferentfromthereadability
Inappropriate 1(5.9%) 1(3.6%) 2(2.2%)
wordusage estimatesofmanuscriptsrejectedforpublication.Forexample,
Lackoftransitions 1(5.9%) 1(3.6%) 2(2.2%) themeanFleschREscorewas28.04foracceptedmanuscripts
(i.e., within the 1–30 range) and 30.77 (i.e., outside the 1–30
range) for rejected manuscripts. Thus, manuscripts that were
contained one ormore link words/phrases thatwere classified acceptedforpublicationcontainedtextthatwasmoredifficult
as add information/provide similarity were 1.75 (95% confi- than was the text in manuscripts that were rejected. The find-
denceinterval[CI]¼1.09,2.79)timeslesslikelytoberejected ingsofOnwuegbuzieetal.(2013),Gazni(2011),andMetoyer-
thanweretheircounterparts,manuscriptsthatcontainedoneor Duran (1993)regarding the predictability of thereadability of
morelinkwords/phrasesthatwereclassifiedasnarrationwere manuscripts make the writing issues meta-theme particularly
1.32(95%CI¼1.01,2.31)timeslesslikelytoberejectedthan noteworthy.
weretheircounterparts,andmanuscriptsthatcontainedoneor
more link words/phrases that were classified as provide an Meta-theme 4: Lack of transparency. This meta-theme also
emphasis were 1.75 (95% CI ¼ 1.07, 2.86) times less likely echoes the reporting criteria as described by the authors of
toberejectedthanweretheircounterparts.Table4displaysthe AERA’s(2006)seminaldocument,whereinitisstipulatedthat
frequency and prevalence rates of themes pertaining to the reports of empirical research should be transparent inasmuch
writingissuesmeta-themeextractedfromthereviewers’com- as reporting should make explicit the logic of inquiry and
mentstothe45submittedmanuscriptsforallreviewers,andas activities that led from the development of the initial interest,
afunctionoftypeofreviewer.Itcanbeseenfromthistablethat topic, problem, or research question; through the definition,
thelack ofclarity theme wasbyfar the mostprevalent. collection, and analysis of data or empirical evidence; to the
Thesixthemesrepresentingthewritingissuesmeta-theme, articulated outcomes of the study (AERA, 2006, p. 33).
each play a role in adversely affecting the readability of a According to the standards, alongside being warranted,
manuscript.Interestingly,Onwuegbuzie,Mallette,Hwang,and ‘‘Reporting that takes these principles into account permits
Slate(2013)providedevidencethatreadabilityplaysanimpor- scholarstounderstandoneanother’swork,preparesthatwork
tant role with regard to the quality of manuscripts. Using the for public scrutiny, and enables others to use that work’’
Flesch Reading Ease (RE) and Flesch-Kincaid Grade Level (AERA, 2006, p. 33). In the context of the IJQM reviewers,
(GL)—two commonly used and easily accessible (e.g., via the lack of transparency meta-theme refers to any criticism
MicrosoftWord)readabilityformulas,amongmanyotherfind- made by a reviewer regarding missing information about the
ings,Onwuegbuzieetal.(2013)observedthat(a)manuscripts methodsusedsuchaslackofinformationregardingthesample
with Flesch RE scores between 0 and 30 are 1.64 more times size, sampling scheme, research design, elements of the data
lesslikely tobe rejected than aremanuscripts with Flesch RE collection process, and/or the analysis process. This meta-
scores greater than 30, and (b) manuscripts with Flesch- themecontainedthefollowingsixthemes:(a)insufficientpro-
Kincaid GL scores of 16 and above are 4.55 times less likely cedures, (b) lack of sampling clarity, (c) lack of conclusion,
to be rejected than are manuscripts with Flesch-Kincaid GL (d) insufficient explanation of analysis, (e) poor discussion of
scores less than 16. Interestingly, with regard to the Flesch table/figure, and (f) did not provide directions for future
RE scores, the findings of Gazni (2011) and Metoyer-Duran research. Specifically, comments relating to insufficient pro-
(1993) were remarkably consistent with those results reported cedurestheme,asthelabelsuggests,occurredwhentheauthor
by Onwuegbuzie et al. (2013). Specifically, Gazni (2011) did not provide adequate procedural information for the
examined the relationship between readability estimates (i.e., reviewer to assess what had been undertaken. Similarly, the
Flesch RE scores of abstracts only) and citation rates for arti- lackofsamplingclaritythemeemergedwhentheauthorfailed
cles published between 2000 and 2009 from the five institu- toprovidesufficientinformationregardingthesamplesizeand
tions(e.g.,Harvard)thatsecurethelargestnumberofcitations. samplingschemeforallphasesofthemixedmethodsresearch
Basedonananalysisofapproximately260,000abstracts,span- study(i.e.,typeofrandomsamplingscheme[e.g.,simpleran-
ning 22 disciplines, Gazni derived a statistically significant, dom sampling] or type of purposive sampling scheme [e.g.,
negativerelationshipbetweentextdifficultyandcitationrates; convenience sampling, criterion sampling]; see, for e.g.
inotherwords,themoredifficultthetext,themoreitwascited. Collins, Onwuegbuzie, & Jiao, 2007; Kemper, Stringfield, &
Additionally, the Flesch RE scores, which ranged from an Teddlie, 2003; Onwuegbuzie & Collins, 2007; Teddlie & Yu,
8 International Journal of Qualitative Methods
Table 5. Meta-Theme 4: Lack of Transparency: Themes Extracted models)toframetheirstudies.Eachofthemorethan40books
FromReviewers’Comments. publishedonmixedmethodsresearchhasauniqueframework
that could have been used by the authors, as do numerous
Nonmixed
journalarticlesandbookchapters.Forexample,betweenthem,
MixedMethods Method Total
Reviewers Reviewers Reviewers the first edition (Tashakkori & Teddlie, 2003) and the second
Theme (n¼17) (n¼28) (N¼45) edition (Tashakkori & Teddlie, 2010) of the Handbook of
Mixed Methods Research contained typologies for several
Insufficientprocedures 2(11.8%) 4(14.3%) 6(13.3%)
stages of the mixed methods research process, including the
Lackofsamplingclarity 1(5.9%) 3(10.7%) 4(8.9%)
purpose (Newman, Ridenour, Newman, & DeMarco, 2003),
Lackofconclusion 4(23.5%) 0(0%) 4(8.9%)
research questions (Plano Clark & Badiee, 2010), research
Insufficientexplanationof 1(5.9%) 1(3.6%) 2(4.4%)
analysis design (e.g., Nastasi, Hitchcock, & Brown, 2010), and data
Poordiscussionoftable/ 0(0%) 1(3.6%) 1(2.2%) analysis(e.g.,Onwuegbuzie&Combs,2010).Yet,theauthors
figure who were flagged by the reviewers did not use any of these
Didnotprovidedirections 1(5.9%) 0(0%) 1(2.2%) frameworks. Nor did they develop their own framework. Fur-
forfutureresearch
ther, three sets of authors’ manuscripts contained a lack of
validity/legitimation discussion about their findings, despite
the fact that there are several mixed methods-based validity/
2007)and/ortherelationshipbetweenthequantitativeandqua-
legitimation frameworks in existence. Indeed, Heyvaert,
litativesamples(i.e.,mixedsamplingdesign;Onwuegbuzie&
Hannes, Maes, and Onghena (2013) identified the following
Collins,2007). The lackof conclusion theme, which, interest-
13 validity frameworks that they called critical appraisal fra-
ingly, was only pointed out by the mixed methods reviewers
meworks (CAFs): Alborz and McNally (2004); Bryman,
(i.e.,n¼4),arosewhenanauthordidnotprovideaconclusion
Becker,andSempik(2008);CaracelliandRiggin(1994);Cres-
totheirstudyortotheirconceptual/theoretical/methodological
wellandPlanoClark(2007;seealsoCreswell&PlanoClark,
essay.Onthetwooccasionswhenreviewerscriticizedamanu-
2011);DellingerandLeech(2007);Dyba,Dingsøyr,andHans-
scriptascontaininganinsufficientexplanationofanalysis,the
sen (2007); Greene (2007); O’Cathain, Murphy, and Nicholl
authorsmadeonlyscantmentionoftheanalysisused.Assuch,
(2008); Onwuegbuzie and Johnson (2006); Pluye, Gagnon,
it was not possible for these reviewers to assess the appropri-
Griffiths, and Johnson-Lafleur, (2009); Pluye, Grad, Duni-
atenessoftheanalysistechniquesused.Oneoftheauthorsdid
kowski, and Stephenson (2005); Sale and Brazil (2004); and
notevenrefertothetablesandfigures,letalonediscussthem,
TeddlieandTashakkori(2009).Fromthese13CAFs,Heyvaert
therebyleavingitforthereaderstointerpretthesevisualrepre-
et al. (2013) generated the following 13 headings that group
sentations purely by themselves, which, in turn, affected the
similarcriteria:criteriaforqualitativepartofthestudy;criteria
clarity of the results that were presented. Thus, this error of
for quantitative part of the study; criteria for mixing and inte-
omissionwasclassifiedunderthepoordiscussionoftable/fig-
grationofmethods;rationaleformixingmethodsstated;theo-
uretheme.AsnotedbytheauthorsofAPA(2010),‘‘Inthetext,
retical framework; research aims and questions; design;
refertoeverytableandtellthereaderwhattolookfor.Discuss
samplinganddatacollection;dataanalysis;interpretation,con-
only the table’s highlights...’’ (p. 130). Finally, the did not
clusions, inferences, and implications; context; impact of
provide directions for future research theme, which arose on
investigator; and transparency. This points to the lack of gen-
oneoccasion,emergedwhentheauthordidnotprovidereaders
eral awareness ofqualitycriteria formixed methods research.
withanyrecommendationsforfollow-upstudiesinthediscus-
Theinappropriateresearchdesigntheme,asthelabelsug-
sion section. Table 5 displays the frequency and prevalence
gests, reflects the two sets of authors who did not specify a
rates of themes pertaining to the lack of transparency meta-
design that was consistent or compatible with the underlying
theme extracted from the reviewers’ comments to the 45 sub-
researchquestionsandproceduresused.Incontrast,thelackof
mittedmanuscriptsforallreviewersandasafunctionoftypeof
application theme referred to a conceptual essay in which the
reviewer.
authorsdidnotillustratetheapplicationsorimplementationof
theframeworkused.Finally,thelackoflinkingresearchques-
Meta-theme 5: Lack of integration. This meta-theme was perti- tion(s)toresearchdesignthemerepresentsasetofauthorswho
nentwhentheauthordidnot(sufficiently) integratethequan- didnotshowtheconnectionbetweentheresearchquestionand
titative and qualitative components but, instead, presented the selectedresearchdesign—forexample,asoutlinedbyOnwueg-
quantitative and qualitative components ina separate manner. buzie and Leech (2006) and Plano Clark and Badiee (2010).
Thismeta-themecontainedthefollowingfivethemes:(a)lack Table 6 displays the frequency and prevalence rates of themes
of mixed methods framework, (b) lack of validity/legitimation pertainingtothelackofintegrationmeta-themeextractedfrom
discussion,(c)inappropriateresearchdesign,(d)lackofappli- thereviewers’commentstothe45submittedmanuscriptsforall
cation,and(e)lackoflinkingresearchquestion(s)toresearch reviewersandasafunctionofthetypeofreviewer.
design.Thelackofmixedmethodsframeworkthemedealtwith
theauthornotusingoneofthenumerousexistingmixedmeth- Meta-theme 6: Philosophical issues. This meta-theme was perti-
ods research frameworks (e.g., designs, typologies, and nent when the author did not (adequately) discuss the
Onwuegbuzie and Poth 9
Table6.Meta-Theme5:LackofIntegration:ThemesExtractedFrom Table 7. Meta-Theme 6: Philosophical Issues: Themes Extracted
Reviewers’Comments. FromReviewers’Comments.
Mixed Nonmixed Nonmixed
Methods Method Total MixedMethods Method Total
Reviewers Reviewers Reviewers Reviewers Reviewers Reviewers
Theme (n¼17) (n¼28) (N¼45) Theme (n¼17) (n¼28) (N¼45)
Lackofmixedmethods 4(23.5%) 2(7.1%) 6(13.3%) Mislabelingofphilosophy 1(5.9%) 2(7.1%) 3(6.7%)
framework Noclearspecificationof 2(11.8%) 1(3.6%) 3(6.7%)
Lackofvalidity/legitimation 0(0%) 3(10.7%) 3(6.7%) self-philosophy
discussion
Inappropriateresearchdesign 1(5.9%) 1(3.6%) 2(4.4%)
Lackofapplication 1(5.9%) 0(0%) 1(2.2%)
Table 8. Meta-Theme 7: Strengths: Themes Extracted From
Lackoflinkingresearch 0(0%) 1(3.6%) 1(2.2%)
Reviewers’Comments.
question(s)toresearchdesign
Nonmixed
MixedMethods Method Total
Reviewers Reviewers Reviewers
underlying research philosophy. This meta-theme contained
Theme (n¼17) (n¼28) (N¼45)
the following two themes: (a) mislabeling of philosophy and
(b)noclearspecificationofself-philosophy.Themislabelingof Clearwriting 6(35.3%) 9(32.1%) 15(33.3%)
philosophy theme refers to the author not showing adequate Enhancequalitativeresearch 3(17.6%) 1(3.6%) 4(8.9%)
knowledge and understanding of the underlying research phi- Usefulframework 1(5.9%) 3(10.7%) 4(8.9%)
Wellgrounded 1(5.9%) 3(10.7%) 4(8.9%)
losophy.Indeed,onereviewernotedtotheauthorthatheorshe
Significant 0(0%) 3(10.7%) 3(6.7%)
ortheyclearlylackedknowledgeofmentalmodelsassociated
Evidenceprovided 1(5.9%) 0(0%) 1(2.2%)
withmixedmethodsresearch.Anotherauthorwascriticizedfor
Identificationofappropriate 0(0%) 1(3.6%) 1(2.2%)
use of the term ‘‘positivism’’ instead of ‘‘postpositivism.’’ philosophy
Now, there are two major types of positivism: (a) classical Thoroughanalysis 0(0%) 1(3.6%) 1(2.2%)
positivism, which was introduced by Auguste Comte (French Welldesigned 0(0%) 1(3.6%) 1(2.2%)
philosopher)and(b)logicalpositivism,whichoriginatedinthe Creative 1(5.9%) 0(0%) 1(2.2%)
ViennaCircle,agroupofEuropeanscholars,duringthe1920s
and1930s.AsstatedbyYu(2003,p.9),‘‘Whenmanyauthors Thenoclearspecificationofself-philosophythemecameto
discusstherelationshipbetweenpositivismandresearchmeth- theforewhenthereviewercriticizedtheauthor(s)fornotmak-
odology,thecontextissituatedinlogicalpositivismratherthan ingclearher orhisortheir ownresearchphilosophy.Another
classical positivism.’’ According to Yu (2003), classical posi- criticismfallingunderthisthemewastheauthor(s)providinga
tivismrepresentsasinglemovement,characterizedbyaphilo- researchphilosophythatcontradictedotherstatementsmadein
sophythatscientificinquiryshouldbeempirical,whichyielded the manuscript. The third and final criticism surrounded the
antirealism and instrumentalism. In contrast, Hacking (1983) author(s) seemingly unwittingly shifting her or his or their
identifiedthefollowingsixmajorthemesoflogicalpositivism: philosophicalpositionduringthemanuscript.Table7displays
(a)emphasisonverification,(b)proobservation,(c)anticause, thefrequencyandprevalenceratesofthemespertainingtothe
(d) downplaying explanation, (e) antitheoretical entities, and philosophicalissuesmeta-themeextractedfromthereviewers’
(f) antimeta-physics. Indeed, logical positivism was discre- comments to the 45 submitted manuscripts for all reviewers
dited as a viable research philosophy after the Second andasa functionoftype ofreviewer.
World War. Thus, when authors refer to logical positivists,
it is very likely that this reference represents a mischarac-
Meta-Themes That Represented a Strength
terization of contemporary quantitative researchers—as was
the case here. Rather, a significant proportion of quantita- The constant comparison analysis also yielded 10 themes that
tive researchers has postpositivistic leanings, whose ontol- each represented a strength of the manuscripts: clear writing,
ogy (i.e., nature of reality) is that understanding of reality is enhance qualitative research, useful framework, well
constructed; epistemology (i.e., nature of knowing) is that grounded, significant, evidence provided, identification of
findings are probably objectively obtained using primarily appropriatephilosophy,thoroughanalysis,welldesigned,and
quantitative methods; axiology (i.e., role of values in creative.Becauseallofthesethemesrepresenttheoppositeof
inquiry) is that research is influenced by values of research- one of the themes (i.e., what we call a theme antonym) dis-
ers; rhetoric (i.e., language of research) is that a formal cussed in previous sections—for example, the enhance quali-
writing style using an impersonal voice predominates; and tative research theme represents the opposite of the did not
methodology (i.e., process of research) that stems from a advancequalitativeresearchthemeunderthelackofjustifica-
deductive logic in which research is influenced by theory/ tion meta-theme—we assume that these 10 strengths-based
hypothesis (cf. Johnson & Onwuegbuzie, 2004). themes are self-explanatory and do not need further
10 International Journal of Qualitative Methods
explication. Table 8 displays the frequency and prevalence lack of warrantedness (i.e., containing 7 themes), lack of
rates pertaining to the themes extracted from the reviewers’ justification (i.e., containing 6 themes), writing issues (i.e.,
comments to the 45 submitted manuscripts for all reviewers containing 6 themes), lack of transparency (i.e., containing
andasafunctionoftypeofreviewer.Itcanbeseenfromthis 6themes),lackofintegration(i.e.,containing5themes),and
tablethatclarityinwritingwasbyfarthemostcommonreason philosophicalissues(i.e.,containing2themes).Theutilityof
for a manuscript receiving praise, just as it yielded a very thisfindingisthatthesesixmeta-themesandtheirassociated
commonreasonforamanuscripttobecriticizedifthisclarity 32themesmaybeusedtoinformreviewersheetsthatcontain
wasabsent. criteria used by reviewers to assess mixed methods research
manuscriptssubmittedtojournalsforreviewforpossiblepub-
lication. This emergent thematic structure could lead to the
Discussion development of up to 32 assessment items for reviewers of
mixed methods research manuscripts such as the reviewer
Learningtoreviewjournalmanuscriptsrepresentsaself-guided
sheet that we have developed from this structure and have
journey, typically involving minimal mentoring and support,
presented in the appendices (see Appendix A for partial
withlittleornoexplicitguidelines(Lu,2012).Despitethefact
reviewer sheet and Appendix B for a list of the 32 items).
that a significant proportion of academic faculty members for-
It can be seen from this reviewer sheet that all 32 items are
mally review journal manuscripts, and despite the increasing
stated in a positive direction as a means of rewarding
bodyofliteraturedescribingaspectsofthepeer-reviewprocess,
authors for containing these elements (i.e., positive reinfor-
itissurprisingthatfewstandardsandcriteriahaveemergedthat
cement) rather than penalizing authors for omitting these
characterize an optimal review. Consistent with our assertion
components (i.e., punishment). We believe that this
here,Lu(2012)madethefollowingobservation:
reviewer sheet not only is potentially useful for reviewers
by providing them with explicit items that characterize a
‘What a good review is’ is not an easy question to answer,
quality manuscript to evaluate as well as for their editors
considering in the literature there is a lack of clarity among
who, subsequently, would be the recipients of quality
scholars in articulating ‘what high-quality or good research
reviews but also is potentially useful for authors of mixed
might be’. As a management journal editor claimed, peer
methods research manuscripts by providing them with expli-
review is inherently subjective because judgements about
cit guidelines for developing these manuscripts. Further, we
knowledge are filtered through a personal lens, which ‘alters
believe that such a reviewer sheet would be helpful for
individualreferees’understandingandshapestheirthinkingin
college-level instructors of mixed methods research courses,
an idiosyncratic fashion’. However, we know only snippets
mentors, advisors, thesis/dissertation chairs/supervisors and
about quality judgements; for example, there is high agree-
ment between reviewers when it comes to rejection and in other committee members, as well as authors of future mixed
identifying a paper of high quality. But such studies rarely methods textbooks and other mixed methods works, and even
wentdeeper.(p.56) writers of future editions of style guides such as the APA
Publication Manual. Importantly, we expect that this reviewer
And as few works as have emerged to date that explicate sheet also would be helpful for mixed methods practitioners
what makes a quality review of quantitative research manu- who can be defined as those ‘‘with theoretical and practical
scripts or what makes a quality review of qualitative research knowledge of three methodologies (i.e., qualitative, quantita-
manuscripts, to date, there have been no works in which the tive, and mixed methods)’’ (Poth, 2012, p. 315).
author has outlined what makes a quality review of mixed Although our reviewer sheet stems from the voices of our
methodsresearchmanuscripts.Yet,amajorfindingfromLu’s 45 talented reviewers, and these voices indubitably have led
(2012) mixed methods research study of 44 experienced to what we deem as 16 quality published articles across our
reviewerswasthattheseparticipantsemphasizedtheimportant two IJQM special issues, this reviewer sheet (and any other
rolethatjournaleditorscanplayinmotivatinggoodreviewing sheets that subsequently are developed from the present or
by specifying explicitly what a good review is. Thus, the pri- future findings) should be subjected to validation/legitima-
mary goal of the current editorial has been to begin the con- tion studies. In particular, future research is needed to deter-
versation among members of the mixed methods research mine the quality of reviews that our newly developed
communityregardingwhatmakesaqualityreviewbyprovid- reviewer sheet generates. Once validated/legitimated, we
inganevidence-basedframeworkforcomprehensivelyreview- contend that such an evidence-based reviewer sheet should
ingmixedmethods research manuscripts. make it easier for reviewers to provide what Fischer (2011,
As can be seen from the findings from the MMCS pre- p. 227) refers to as value-added reviews. Such improvement
sented in this editorial, together, the reviewers offered the in quality of reviews, in turn, would improve the quality of
authorsfeedbackthatwasgreaterthanthesumoftheindivid- mixed methods research articles. And little is more crucial
ual reviews—for which we are extremely grateful to the to the advancement of both science and the field of mixed
reviewers. The major finding yielded from these 45 special methods research than the establishment of optimal reviews
issue reviewers is the emergence of the following six meta- via the information gatekeepers—in this case, reviewers of
themes (and 32 themes) that characterized their criticisms: mixed methods manuscripts.