ERIC EJ859463: A Case Study of a Community-Based Participatory Evaluation Research (CBPER) Project: Reflections on Promising Practices and Shortcomings

2009·0.11 MB·English

by  ERIC

MichiganJournalofCommunityServiceLearning Spring2009,pp.34-47
A Case Study of a Community-Based Participatory
Evaluation Research (CBPER) Project:
Reflections on Promising Practices and Shortcomings
JiniPumaUniversityofColorado,Denver
LaurieBennettandNickCutforthUniversityofDenver
ChrisTombariSpringInstitute
PaulSteinColoradoRefugeeServicesProgram
Thisinstrumentalcasestudydocumentsacommunity-basedparticipatoryevaluationresearch(CBPER)pro-
jectthatinvolvedacommunitypartner,twograduatestudents,afacultymember,andanexternalfunder.It
highlights the fact that a participatory evaluation model is a viable way to conduct community-based
research(CBR)whenacommunityorganizationneedstoknowiftheprogramservicestheyareofferingare
effective.Theidentificationofthepromisingaspectsandshortcomingsofthisprojectadvancethetheoreti-
calandmethodologicalunderstandingsoftheCBRfieldandpromotesbetterCBRpractice.
C
ommunity-basedresearch(CBR)iscollaborative, Inthesecases,programevaluationcanbecomepartof
change-oriented research that engages faculty mem- theCBRparadigm.Evaluationresearchisdefinedas
bers, students, and community members in projects the systematic application of social research proce-
addressing community-identified needs (Strand, duresinassessingsocialinterventionprograms(Rossi
Marullo,Cutforth,Stoecker,&Donohue,2003).With &Freeman,1993).Community-basedresearcherscan
itsrootsinactionresearch,participatoryresearch(for use evaluation models such as participatory evalua-
adetaileddescriptionbetweenthetwo,seeBrown& tion(Patton,1997a;Stoecker,1999)orempowerment
Tandon, 1983), and popular education, CBR is an evaluation (Fetterman, 1994a; Fetterman, 1994b;
importanttoolinengaginginstitutionsofhigheredu- Stoecker) to help guide the community organization
cationwithlocalcommunities(seeStrandetal.,2003 throughtheevaluationprocess.TheCBRprojectthat
formoreonthehistoryofCBRanditsrelationshipto isdescribedinthisarticlealignsmorewiththepartic-
thethreestatedmodels).Indeed,severalacademicdis- ipatoryevaluationmodel,whichisreallyanextension
ciplines, including education (Stocking & Cutforth, of the stakeholder-based model with a focus on
2006), environmental health sciences (O’Fallon & enhancing evaluation utilization through the primary
Dearry, 2002), international education (Lewis & user(s)’increasedparticipationintheresearchprocess
Niesenbaum, 2005), nursing (Kelley, 1995), occupa- (Cousins&Earl,1992).Theprinciplesofparticipato-
tionaltherapy(Taylor,Braveman,&Hammel,2004), ry evaluation can be summarized as follows: (a)
planning(Reardon,1998),publichealth(Israel,Eng, involve participants at every stage of the research
Schulz, & Parker, 2005; Minkler & Wallenstein, process;(b)makesuretheparticipantsowntheevalu-
2003),socialwork(Rogge&Rocha,2004),andsoci- ation;(c)focustheprocessontheoutcomesthepar-
ology (Buroway, 2005) have acknowledged CBR’s ticipantsthinkareimportant;(d)facilitateparticipants
contribution to the paradigm shift toward university- toworkcollectively;(e)organizetheevaluationtobe
community involvement. Furthermore, the growing understandableandmeaningfultoall;(f)usetheeval-
recognition of the potential of CBR to address com- uationtosupportparticipants’accountabilitytothem-
plexcommunityneedsisrecognizedbytheinsistence selvesandtheircommunityfirstandoutsiderssecond,
on forming CBR partnerships contained in grant ifatall;(g)developtheevaluatorroleasafacilitator,
guidelines from federal agencies (e.g., the U.S. collaborator,andlearningresearcher;(h)developpar-
Department of Housing and Urban Development’s ticipants’rolesasdecisionmakersandevaluators;(i)
Community Outreach Partnership and the Center for recognize and value participants’expertise and help
Disease Control’s Prevention Research Centers them to do the same; and (j) minimize status differ-
Program)andotherfundingsources. ences between the evaluation facilitator and partici-
WithCBR,theresearchissueandquestionoriginate pants(Patton;Stoecker).
withthecommunity.Oftentimes,communityorgani- Researchthatisfirmlybasedinthecommunityis
zationswanttoknowiftheirprogramsare“working.” an important component of the community engage-
34
ACaseStudyofaCommunity-BasedParticipatoryEvaluationResearch(CBPER)Project
ment movement and with increased awareness, the busiest avenues of the city. The painstakingly
participatory evaluation approach can be a vehicle restoredoldhouseisthehomeoftheSpringInstitute
through which communities can become further and its WorkStyles program. WorkStyles is a two-
engaged in the programs which serve them. The week, intensive, pre-employment training class for
practical challenges of conducting research in and newlyarrivedrefugees.Theoccasionforthisvisitis
withacommunity,suchasunrealisticgoalsandtime- a graduation ceremony: the 188th class of
frames,minimalresources,alackofpersonalinvest- WorkStylesisjustending,anditistimetocelebrate.
ment,andmethodologicalchallengesassociatedwith Oneofthe visitorswrites:
recruiting participants (Stocking & Cutforth, 2006;
I am greeted at the door by a mix of invited
Strand et al., 2003; Willis, Peresie, Waldref, &
guests,SpringInstitutemanagementandstaff,
Stockmann,2003),canmakeanyCBRprojectdiffi-
and some of the students from the graduating
cult to execute and complete; a CBR study whose
class.The atmosphere… is one of excitement
purpose is to evaluate a program serving that com- and anticipation. I can feel that it is a special
munityisnodifferent.Throughaninstrumentalcase day,notonlyforthestudentsbutalsoforthose
study design1 (Merrian, 1988; Yin, 2003), we who make all they do here possible.A warm
describeandinterpretaseriesofstepsandsequence afternoonlightfillsthefoyerandIammadeto
ofactivitiesthatledtoaparticularcommunity-based feelrightathomebyallwhowelcomeme….
Peopleofmanyracesandnationalitiesmin-
participatoryevaluation research(CBPER)project’s
gletogetherasweallbegintotakeourseats.I
successandfailure.Specifically,weaddressthefol-
feelthesenseofbeingpartofsomethinglarge
lowing two questions: first, what worked and what
and hopeful.An older gentleman from Sierra
did not with this CBPER project, and second, what
Leone who is one of the graduates stands in
lessonscanbelearnedfromapromising,butimper- front of the full room and greets guests and
fect, CBPER experience that might be extended to graduates as the ceremony begins. He is
CBR generally? By addressing these questions, the dressedinbeautifulAfricanrobesandcapand
case study contributes to the dialogue in the field he seems to carry the dignity of his country-
regardingbestpracticesofCBR. men with ease and comfort.After welcoming
This article contains six sections. The first section all, he opens the ceremony with a prayer and
…lay[s]outtheafternoon’sevents.
introduces,inanarrativefashion,theCBPERproject
which is the subject of this case study; describes the As the graduation ceremony begins, the observer
project’s primary stakeholders; and documents how continues:
the project unfolded (including how stakeholders
shaped its conception and design, collection and First,comestheintroductionofallthegradu-
ates.Astheystandonebyone,theytellalittle
analysisofdata,andthedisseminationoftheresults).
aboutthemselvesandtheircountriesoforigin:
Thesecondsectiondiscussestheoverallimpactofthis
Burma, Eritrea, Vietnam, Sierra Leone, and
project.Thethirdsectionconsiderspromisingaspects
more. One by one they thank those at the
ofthisCBPERproject,whilethefourthsectioncon-
Spring Institute who made this course possi-
sidersshortcomingswiththeproject.Thefifthsection
ble. Their English proficiency varies, but not
identifieslessonslearnedforfutureresearch,andthe the gratitude they share with those in atten-
sixth section discusses the implications of this case dance. Blessings are offered by many, to the
studytothefieldofCBR.Ingeneral,thiscasestudy institute,theinstructors,andtoallthoseseated
showshowopendialoguecanoccurwhenacademics infrontofthem.
andcommunitypartnerslistentoandvaluethepartic- Next[comes]entertainmentprovidedbythe
graduates…especially moving is a perfor-
ularexpertiseeachbringstotheproject.Theresultis
mance by a gentleman from Vietnam. He
the creation of a level playing field, which enables
greets the crowd and explains in broken
eachpartnertobefullparticipantsineachphaseofthe
English that he spent 14 years in a refugee
work.Theidentificationofthefactorsthatcontributed
camp in the Philippines before arriving in
to the success and failure of this project will inform America 2 months ago. He says he wants to
professorsandstudentsastheylaunchtheirownCBR sing a song he wrote about what he had been
projects,andthus,enrichstudents’educationandserve through to get toAmerica.A guitar is passed
theneedsofthecommunity. forward,andashebegins,Icanfeeltheperse-
verance and pride that sustained him through
The Project hisordeal.Iamalmostmovedtotears.
Narrative Soonafter,allinattendancewhohaveyetto
introduce themselves do so. There are people
Visitors approach an oldVictorian house, nestled fromothernonprofitorganizations,priorgrad-
into a side street, just blocks away from one of the uatesfromtheinstitute,institutestaffandman-
35
Pumaetal.
agement,aswellasindividualswhohavepro- Stakeholders
vided funding over the years…I feel honored
tobeamongstagroupofpeoplewhogiveso The authors of this article represent the primary
much to better the lives of others. Although stakeholders(seeFigure1),whoaredescribedbelow.
attendees are of many differing races and
TheCommunityPartners
nationalities,aprideandunityfilltheroomas
theyshareinthecelebrationofthegraduates. CRSP. The Colorado Refugee Services Program
Finally,thediplomasarepassedoutandafew (CRSP) is the Division of Refugee Services in the
wordsofclosingareofferedbeforewealljoin
Office of Self Sufficiency of the Colorado
together in a song that marks the end of the
DepartmentofHumanServices.ThegoalofCRSPis
ceremony. It is time to enjoy the potluck of
toachieveeffectiveresettlementandrapidself-suffi-
foodandfortune…providedbythegraduates
ciencyforallrefugeesinColorado.CRSPisrespon-
ofthe188thWorkStylesclass.
sible for the submission and implementation of
This“visitor”isactuallyaresearcherwhopartici- Colorado’s State Plan for refugee resettlement,
patedinaCBRstudy;theabovepassageisexcerpt- whichisapprovedandfundedbythefederalOffice
ed,withminimalchanges,fromhisfieldnotesonthe ofRefugeeResettlement.CRSPprovidesstatelead-
lastdayofhisobservation.Thecentralpurposeofthe ership, networking, consultation, supervision, and
study was to evaluate the impact of theWorkStyles monitoringonbehalfofthemanypublicandprivate
programupontheshort-termemploymentofitsgrad- agencies in Colorado that serve refugees, either as
uates. The Colorado Refugee Services Program partofaspecificcaseloadfundedbycontractswith
(CRSP)—themainfunderofWorkStyles—requested CRSP, or as part of a larger mainstream caseload
andfundedtheevaluationstudy.Thestudywascon- fundedbyothersources.Theauthorityforthework
ducted under the auspices of the Colorado ofCRSPisfoundintheRefugeeActof1980.
Community Based Research Network (CCBRN), The Spring Institute for Intercultural Learning.
housed at the University of Denver. The actual The Spring Institute is a Denver-based nonprofit
researchers—the observers who created field notes foundedin1979topromoteinterculturalsharingand
(such as those excerpted above), the interviewers communicationasacontributiontoamorepeaceful
who spoke to program participants, trainers, and world. Their principal goal is to demonstrate that
communityagencies,andthequantitativeresearcher national, cultural, linguistic, and ethnic differences
whoanalyzedemploymentdataofrefugeeswhopar- areassetsthatfosterunderstandingandcooperation.
ticipatedinWorkStylesascomparedwiththosewho TheSpringInstituteprovidesdirectservices,techni-
did not—were five University of Denver graduate cal assistance, training, and consulting services
students.Twoofthestudents(co-authorsofthisarti- regionally, nationally, and internationally. Programs
cle)tooktheleadindesigningandimplementingthe includeEnglishLanguageTrainingindifferentbusi-
qualitative and quantitative portions of the study, ness and community settings: Employability
under the general supervision of a professor (also a Services;Intercultural,Cross-cultural,andDiversity
co-author)specializinginCBR. Training; Interpretation and Translation Services;
However,thecompositionofthestudyteamextend- CommunityIntegration;andInterpreterTraining.
edbeyondtheUniversity.TheDirectorofCRSP(fun- The Spring Institute for Intercultural Learning
derandanotherco-author)playedaroleintheproject started WorkStyles in 1985. Funded through the
byprovidingvaluableinputandfeedback,especially Federal Refugee Social Services and Targeted
at the beginning and end stages of the project.And, Assistance Grant programs,WorkStyles is designed
morepivotally,membersoftheWorkStylesstaff,espe- for non-native English speakers, specifically, to
ciallyitsEmploymentTrainingManager(anotherco- address the barriers to learning created by lack of
author)andtrainerswhoarethemselvesrefugeesand confidenceandself-esteem.Todate,ithasmorethan
had participated in the WorkStyles program as new 220 programs. WorkStyles is a two-week, 60-hour
arrivalsintheU.S.,wereintegrallyinvolvedascom- intensive course focusing on pre-employability and
munitypartnersineverystageofthestudy(fromplan- personal effectiveness skills and is designed to help
ning/design,todatacollection,toreviewingandpro- individuals obtain, retain, and become successful in
viding substantive input on draft reports and conclu- employment. Its training staff includes members of
sions). Thus, members of very different communi- the refugee community who have first-hand experi-
ties—a state refugee service program, a university, a enceofthechallengesofenteringintotheunfamiliar
local pre-employment refugee training program, vol- U.S.workforceculture,andwhothemselvesgradu-
untary agencies (VOLAGs) who case manage the atedfromtheWorkStylesprogram.Theoverallpro-
refugees, and the refugees themselves—collaborated gram includes developing resumes, completing
toproduceaCBPERstudy. applicationforms,practicinginterviews,andmaking
36
ACaseStudyofaCommunity-BasedParticipatoryEvaluationResearch(CBPER)Project
phonecallsaboutjobs,aswellassettinggoals,iden- Education at the University of Denver (DU). The
tifyingskills,understandingAmericanworkculture, CCBRNisaclearinghousethatconnectstheresearch
and solving problems on the job. WorkStyles needsofcommunitypartners(i.e.,schools,nonprofits,
employs a variety of strategies which help to rein- socialagencies,andcommunity-basedorganizations)
forceself-confidencebyencouragingpeopletoshare with the skills and interests of university researchers
their knowledge and experience: videotaped role (i.e.,professorsandgraduatestudents)(Stoeckeretal.,
plays, brainstorming activities, skits to demonstrate 2003). Funding from local and national foundations
cross-cultural situations, small-group problem solv- andstateagenciesenablesgraduatestudentsandpro-
ing, and individual exercises (Michienzi, 2003).As fessors to conduct up to a dozen CBR projects each
peopleparticipateintheseactivities,theytakerisks, year that address community-identified needs in the
andtheygainconfidenceintheirabilitytousetheir fields of education, public health and nutrition, and
verbal English skills in new and unfamiliar situa- environmental justice.The majority of these projects
tions.Theprogramstartswiththeassets,skills,and involvequalitative,quantitativeand/ormixedmethods
talents that refugees bring to the job market in the evaluation of new or existing programs; however,
United States and then moves out into practical needs assessments, oral history projects, and
strategiesrefugeescanuseforsuccessfullyapplying exploratoryresearchstudieshavealsobeenundertak-
those skills to finding and maintaining employment en(fordetailsseewww.ccbrn.org).
andbuildingcareers. These projects provide opportunities for graduate
students to hone their CBR knowledge and skills
TheUniversityPartner
through a pathway of CBR opportunities that
The university partner in this project was the progressfromcourse-based,introductory,andfacul-
Colorado Community-Based Research Network ty-supervised experiences to increasingly sophisti-
(CCBRN) housed in the Morgridge College of cated and self-directed assignments. This pathway
Figure1
TheRelationshipofCommunitiesInvolvedintheCBPERProject
CRSP
Fundingfor
Evaluation
Fundingfor Fundingfor
Programming Programming
DU
Refugees
VOLAGs SpringInstitute CCBRN
CBRPartnership
Services Services
Refugees
37
Pumaetal.
beginswithaclassinwhichstudentsareintroduced author,thecoordinatoroftheCCBRN,tofindoutif
to the principals and practices of CBR and are fundingcouldbeacquiredtocraftameaningfuland
requiredtodesignandimplementaresearchproject rigorous evaluation ofWorkStyles. Somewhat coin-
withacommunitypartner.Studentswishingtodeep- cidentally,andsoonaftertheinitialmeetingbetween
en their CBR experience with community partners WorkStyles and CCBRN, the director of CRSP
can enroll in a CBR practicum class (one quarter), informed the WorkStyles program manager that he
andthenparticipateinapaidCBRinternship(upto wasinterestedinmirroringonasmallscaleamulti-
ayear).Paidresearchopportunities(suchastheone ple site federal study that also was focusing on
described in this article) exist for students with the employment outcomes, to determine if the learning
prerequisite experiences and skills. Finally, students from and use of an evaluation would be less mean-
who are considering CBR as a potential career can ingful because of limited statistical power.Also, as
utilize the approach in master’s theses and doctoral the state agency authorized to develop and imple-
dissertations(Stocking&Cutforth,2006). ment the State Plan for refugee resettlement that is
Aspreviouslymentioned,thereweremembersof approved and funded by its federal partner (the
multiple communities (the refugees, VOLAGS, OfficeofRefugeeResettlement),theCRSPwasina
CRSP, the Spring Institute, and CCBRN) that were positiontousetheinformationgainedfromthissmall
involvedinthisprojectatsomelevel;however,while scaleevaluationtoinfluencepublicpolicyasitper-
recognizing the contributions of all stakeholders in tained to the development, measurement, and posi-
thisevaluationstudy,thisarticlefocusesonthepart- tioning of programs intended to achieve rapid self-
nershipbetweentheSpringInstituteandtheCCBRN sufficiency forrefugees.The CRSP directorwanted
(SeeFigure1)duetoeachpartner’scentralinvolve- toknowifWorkStylescouldgetsuchanevaluation
ment in identifying the research questions, research done with $8,000 of available funding, and the
design,anddatacollectionstools,andthencollecting WorkStyles program manager immediately referred
andanalyzingthedataanddisseminatingtheresults. him to the CCBRN.These preliminary contacts led
toaseriesofmeetings,attendedbyrepresentativesof
ProjectConceptionandDeliberation
CRSP,WorkStyles,andCCBRN,toworkonthepro-
Consistent with the principles of CBR (Strand et ject.CRSP’sfundingwouldenabletheDUgraduate
al.,2003),theinitialseedofanideaforthisevalua- studentsassociatedwithCCBRNtobepaidfortheir
tionprojectoriginatedwithintwoofthecommunity time.Thepurposeofthemeetingswastodecideon
partners: the CRSP and the Spring Institute/ theresearchquestions,measures,andmethods.
WorkStyles. The managers and coordinators of the
ProjectDesign
WorkStyles program had begun to ask themselves
how they could figure out whetherWorkStyles was Inthisseriesofinitialmeetings,aswellasthrough
being successful in its mission. Specifically, they numerous e-mail conversations, these participants
wonderedwhatmeasuresofsuccesstheycoulduse. (partners) crafted a design for the evaluation study
Anecdotal information led them to believe that the withcarefulconsiderationoftheneedsofthefunder
program was indeed successful. For instance, pro- (CRSP), the lead community partner (Spring
gramgraduateswouldcomebackandtalkenthusias- Institute/WorkStyles), and the university (CCBRN/
ticallyabouttheirlives;employerswouldreportpos- DU).Atthisearlystage,thefunderwasfocusedsole-
itive experiences with new hires who had been ly on obtaining a clear picture of WorkStyles’ suc-
through the program; and the VOLAGs who case- cess/failure through quantitative research methods.
managed the refugees continued referring them to Specifically, the funder was interested in answering
WorkStyles without reservation. WorkStyles staff theresearchquestion“DoesWorkStylesimpactshort-
also conducted a survey of past participants, case term employment outcomes for refugees?” To meet
managers,andemployerstounderstandsomeofthe thefunder’sneed,oneoftheleaddoctoralstudentson
positivelong-termbenefitsoftheprogram.Thissur- theproject—inthedualroleofthequantitativeexpert
veyconsistedoftheWorkStylestrainersinterviewing and project manager—collaborated closely with
former students and colleagues; it yielded data that WorkStyles staff to design a quasi-experimental,
warranted further follow-up. However, the Spring matched control group study. A matched control
Institute personnel felt that they lacked not only design “matches” similar subjects from two groups
impartiality,butalsothetechnicalskillstocreateand (WorkStylesandacontrolgroupofrefugeeswhohad
implementspecificevaluativemeasuressothatatrue not attendedWorkStyles) on key attributes.The lead
pictureofthesuccesses(andfailures)ofWorkStyles communitypartner’sexpertise(i.e.,theunderstanding
mightbepresentedtotheoutsideworld. ofwhatfactorsorattributeswereappropriateandfea-
As a first step toward an evaluation, the sibleformatchinggroupsofrefugees)andtheexper-
WorkStyles program manager approached the third tiseoftheuniversitypartner(i.e.,knowledgeofquan-
38
ACaseStudyofaCommunity-BasedParticipatoryEvaluationResearch(CBPER)Project
titativemethodology),dovetailednicelytoproducean evaluation study design was being conceptualized.
initialquantitativestudydesignthatpleasedthefunder The lead qualitative researcher and the WorkStyles
and,inpart,thecommunitypartner. managerwereinvitedtoattendthefirstclassmeeting
The lead community partner, however, had some and“pitch”theirprojecttothestudents.Indoingso,
reservationsfromtheverybeginningaboutwhether theywereabletorecruitthreeadditionalstudentsto
thiskindofstudywould(orcould)presentamean- participateintheintensiveobservationofWorkStyles
ingfulpictureofWorkStyles.TheWorkStyles’man- trainingsessions,toconstructfieldnotes,andtouse
agerbelievedthatitwouldbedifficulttofindanasso- their resulting knowledge to help refine the refugee
ciation between the WorkStyles program and early focus group study design. These students received
employment outcomes. He described his reasoning universitycreditfortheirparticipationinthestudy.In
inthefollowingway: thisway,thecapacitiesoftheresearchteamweresig-
nificantlyexpanded.
One,it’satwo-weekprogramthattakespeople
Thus,amixedmethodsdesignwascollaboratively
outofthejobsearcheventhoughwe’reteach-
created,inanefforttosatisfytherequirementsofthe
ing them job preparation skills… and two we
funder, the lead community partner, and university
don’t have any control over their job place-
partner.Missingfromthetableatthisstageofthecol-
ment…we’rejustatrainingorganization…[job
laboration,unfortunately,wereactualmembersofthe
placement]isnotpartofourfunding.
refugeecommunity;however,atthetimetheresearch
If left to him, the WorkStyles manager would have team felt that their interests were being adequately
chosen long-term employment outcomes for the representedbytheWorkStylesmanager.Movingfor-
study. However, this choice would have required at ward, the collaborative study design constantly
least an 18-month study, and the funder’s require- underwentchangesduetounexpectedstudyscenar-
mentwasthatthestudybecompletedwithinthefis- ios,suchasmissingdatainthestate’sdatabaseand,
cal year, affording only a 9-month time frame. As aswewillseelater,thefailureofparticipantstoshow
such, the community partner concluded that they up to interviews and focus group sessions. In other
needed“someotherdatathatatleastgivessomeevi- words,astheuniversityresearchersandleadcommu-
denceofthe[non-quantifiable]successes.” nitypartnerimmersedthemselvesinthecollectionof
Tothatend,itwasagreedthataqualitativedimen- data,theresearchdesignevolvedandemergedasdic-
sionbeintroducedtotheprojectdesign.Thefunder tated by circumstances. The emergent nature of
was not opposed to this; in fact, he was even researchdesignisfrequentlyarealityofCBR(Strand
intrigued by the possibility. However, he remained etal.,2003).
interested primarily in the quantitative results.
DataCollectionandAnalyses
Nonetheless,thequalitativeportionforgedahead.A
second lead doctoral student with a qualitative Both the quantitative and qualitative lead
researchfocuswasaddedtotheteam,andthestudent researchers,incloseandconstantcollaborationwith
and the WorkStyles manager collaboratively con- the lead community partner, moved forward sepa-
structed a qualitative design. It combined intensive rately, but simultaneously, to collect the necessary
observationofWorkStylestrainingsessionswiththe data. For the quantitative matched-control group
refugees,interviewsofVOLAGrepresentativeswho study,WorkStylesstaff(includingrefugeeswhowere
referred refugees to the WorkStyles program, two formerWorkStyles participants) diligently compiled
focusgroupsoflooselymatchedcomparisongroups descriptivedata(e.g.,aliennumber,name,countryof
of refugees,2 and a final focus group of the origin, date of arrival, gender) month by month for
WorkStylestrainers.Theobjectivesofthequalitative each of the three WorkStyles classes studied and,
methodscomponentofthestudyweretwo-fold:first, working with VOLAG staff, for a control group of
to address the question of whether WorkStyles non-WorkStyles participants (matched on factors
helpedrefugeesobtainearlyemploymentaswellor such as level of English language proficiency and
betterthancomparableprograms;andsecond,topro- date of U.S. arrival). Detailed employment data for
vide a more nuanced picture of the impact the each of the study participants were obtained from a
WorkStyles program was having in aiding refugees CRSP program analyst. (In the process of this data
toadjusttotheirnewlivesandworksituationsinthe collection,suspicionsabouttheincompletenatureof
UnitedStates.Theuniversitypartnerwasabletocon- data regularly collected by CRSP through the
tributetotheseobjectivesnotonlythroughtheefforts VOLAGS were confirmed; thus, the study affirmed
oftheleadqualitativedoctoralstudentresearcher,but that recent improvements in the transfer of employ-
also through a graduate-level CBR class, taught by ment and resettlement data between the VOLAGs
the third author, which was serendipitously sched- andCRSPwereneededtoensuretheintegrityofthe
uled to commence soon after the time at which the stateemploymentdata.)Allthenecessarydatawere
39
Pumaetal.
sent to the lead quantitative doctoral student for refugees to participate in the focus groups. Despite
analysis.Becausethesedatawerecollectedinanon- those staff members’ almost Herculean efforts, a
going fashion, frequent communication via email numberof the planned sessions had to be cancelled
and face-to-face meetings occurred between the atthelastminuteduetotheinabilityofparticipants
WorkStyles staff (including its refugee community to attend. A number of obstacles, which are not
members)andthedoctoralstudent. uncommonwiththerefugeepopulation,butwerenot
Forthequalitativepartofthestudy,fourtypesof anticipated,cameintoplay.Theseincluded:inflexi-
in-depth, qualitative data were ultimately collected: ble employment schedules, lack of transportation,
first,20hoursofobservationoftwoWorkStylespro- lack of child care, and lack of appreciation for the
gramsessions,performedbyCBRgraduatestudents; U.S.culturalexpectationthatappointmentsmadeare
second,interviewsofVOLAGrepresentatives;third, appointments kept. (In subsequent research projects
focus groups of WorkStyles and non-WorkStyles with the refugee community conducted by the first
refugees; and fourth, a focus group of WorkStyles author, these barriers have been addressed).
trainers (including three trainers who are refugee Ultimately, one very rich focus group of employed
community members). In most CBR studies, WorkStyles graduates, and two separate interviews
unplannedeventscanoccur(Strandetal.,2003)and withoneemployednon-WorkStylesrefugee4andone
this study was no exception. In this project, one unemployednon-WorkStylesrefugeewereconduct-
unforeseen occurrence was the observation team’s ed. Some would argue that basing conclusions on
strong recommendation to change the make up of suchasmallsamplesizeisincompleteresearch.This
focusgroups,baseduponwhatithadobserved.The argument reflects the tension between conducting a
initialdesign(seenote2)hadcontemplatedconduct- methodologically sound study and overcoming the
ing only two focus groups. The observation team realitiesofconductingresearchwithandforthecom-
objected to this design, believing that the only way munity.Inthisparticularcase,wewerelimitedinthe
refugees would speak candidly about their training, amountoftimeandresourceswecouldaskourcom-
jobsearch,andemploymentexperienceswouldbeif munity partner to devote to recruiting participants.
four separate focus groups were constituted: Despite this limitation, the focus group and inter-
employed WorkStyles graduates, unemployed viewswererecorded,withtheconsentoftherefugee
WorkStyles graduates, employed non-WorkStyles participants, and transcribed. The lead qualitative
graduates, and unemployed non-WorkStyles gradu- researcher thematically analyzed the transcripts, tri-
ates. Responding to the observation team’s objec- angulatingresultswiththosegleanedfromtheobser-
tions, the lead doctoral student and the WorkStyles vationfieldnotes.
manager revamped the focus group design, and the AfinalfocusgroupwithfiveWorkStylestrainers
graduatestudentteam,usingitsunderstandingfrom (dubbed“trainers’talk”bytheparticipants),threeof
the observations and the themes derived from the whomwererefugeesthemselves,washeldtocapture
fieldnotes,constructedprotocolsforconductingthe theirexperiencewiththeWorkStylesprocess,aswell
focusgroups.Again,itshouldbenotedthattheactu- astomember-checksomepreliminaryinterpretations
al refugee community members of the WorkStyles (seebelow).Thatsessionwasalsorecordedandtran-
staffortheprogramclientsdidnotparticipateinthis scribed,andtheresultstriangulatedwithinformation
studydesigndecision—perhapsamissedopportuni- emerging from the other qualitative data sources
tyforbroadeningthecommunitycollaboration. describedabove.
Next, the rationale behind having qualitative com- Clearly,theunanticipatedproblemswiththesam-
parisongroups(tocomparetheimpactofWorkStyles pling design required a flexible and creative
versus other comparable pre-employment programs) response. Although the team’s response may have
waschallengedbytheinterviewinformationgleaned detracted from the rigor of the research (a tension
fromtheVOLAGrepresentatives.TheVOLAGsmade discussed in the introduction of this article), it
it known that there really were no comparable job ensuredthatdatacollectioncouldbecompletedina
preparationprogramsinthearea,sotheyreferredall timelyandqualitymanner,whichiswhatwasguar-
refugees with the requisite language proficiency to anteedtothefunder.
WorkStyles.Thus,anyhopethatthroughfocusgroups
InterpretationandVerificationofFindings
the researchers would be able to make meaningful
comparisonsbetweentheimpactsofWorkStylesand From an interpretive standpoint, the quantitative
othercomparableprogramswasdashed. results revealed that completing theWorkStyles pro-
Thefocusgroupsthemselvesalsodidnotproceed gram did not significantly affect the short-term
according to the revised plan. In an effort to foster employment outcomes of the participants, as com-
trust with the prospective participants, the refugee pared with the matched-control group. However, the
WorkStyles staff members identified and contacted qualitativefindings,reflectingthecentralthemesthat
40
ACaseStudyofaCommunity-BasedParticipatoryEvaluationResearch(CBPER)Project
emerged from the qualitative data, revealed that Otherwise, she believed that the “whole picture” of
WorkStyles was likely contributing to the long-term WorkStyles’impactscouldnothavebeencommuni-
employmentsuccessofitsgraduates,aswellastotheir cated;shefeltthat“itwouldhavedoneaninjusticeto
overall well-being in their new world. Specifically, theWorkStylesprogram”tohaverestrictedthefinal
three themes (culture, capabilities, and confidence, report to the quantitative portion of the study.
described as “the three C’s” in the final evaluation Interestingly,theincompletenatureofthepictureof
report) emerged. These themes reflect the fact that WorkStylespresentedbythequantitativeresultshad
WorkStyles honors the cultures from which the been anticipated by the community partner, which
refugeescome,aswellasteachesthemaboutthenew wasoneofthereasonswhyamixedmethodsdesign
U.S.cultureinwhichtheynowfindthemselves;that was pursued in the first place. Ultimately, the final
WorkStylesteachespracticalcapabilitiesuponwhich reportwasforwardedtothefunder,andaface-to-face
refugeescanrelyonwhenfindingandkeepingajob; meetingwasheldtodiscussthereportwithhimand
and that WorkStyles instills confidence in refugees alltheevaluationstakeholders.
whichiscriticalforthemtosucceedintheirnewlives.
Impact of the Project
Verification of these results and their interpreta-
tions occurred in two ways. First, the qualitative As stated above, the quantitative results revealed
interpretation(especiallythe“three“Cs”)wasshared thattherewerenostatisticaldifferencesbetweenthe
with the WorkStyles staff -- both refugee and non- WorkStyles group and the control group in early
refugeemembers--attheendofthe“trainers’talk” employment outcomes. This answered the original
focusgroup.Theleadcommunitypartnerenthusias- question posed by the funder (CRSP). However, it
tically endorsed these themes, even to the point of wasconcludedthat90dayswasnotenoughtimeto
incorporatingthemintopresentationmaterialsabout seetheemploymentimpactsoftheWorkStylespro-
WorkStyles to be used at a “Training the Trainer” gram.Thus,alongerstudytimeframewasneeded.5
seminarthefollowingweek.Second,theleadquan- Thefunder’sinitialreactiontothequantitativestudy
titativeandqualitativedoctoralstudentseachcreated results was that one of the WorkStyles grants was
draftreportsrelatingtheirfindings,sharedandedited possibly inappropriate as a funding source because
eachothers’work,andthequantitativedoctoralstu- its intention was to support early employment pro-
dent merged the two into a single draft report.This grams. Discussion ensued regarding this point. The
mergeddraftwassharedwithWorkStylesstaff,and WorkStyles manager pointed out that WorkStyles’
thetwoleadresearchersconductedtwoface-to-face purpose was to assist refugees with retention and
meetings(andwrotemanyemails)toobtainanddis- long-term success, and that as a pre-employment
cuss the staff’s feedback. Two of the refugee staff training program, WorkStyles had no control over
members provided valuable input based upon their refugees’ job placement. That responsibility was
ownexperiencesandobservations.Thefeedbackwas alreadyassignedtotheVOLAGs.
carefullyregarded,andledto,insomecases,signifi- However, after reviewing the results and conclu-
cantchangesinthereport’sobservationsandrecom- sionsofthequalitativepartofthereport,thefunder
mendations. Thus, the experience and expertise of was able to see that there were longer-term integra-
both the university and the lead community partner tionbenefits.Integrationisatopicthatholdsahigh
wereamplyreflectedinthefinalreporttothefunder. priority in nationwide refugee resettlement, and the
reportallowedCRSPtobetterpromoteandposition
DisseminationoftheResults
WorkStyles within the range of beneficial services
Theresultscommunicatedtothefunderwerenei- for refugees. The funder concluded that while the
ther in substance, nor in form, what had originally justification and outcome statements for the
been anticipated. The university partner had under- WorkStylesfundingstreamsmayneedtobealtered,
stoodthatalargelyquantitativereportwouldbepro- therewasastrongneedfortheprogram.Thefunder
duced for the funder, and the qualitative results hadrecently(sixmonthspriortotheissuanceofthe
wouldbemorefortheinternalconsumptionandedi- report)identifiedanadditionalfundingsourcemore
ficationofthecommunitypartner.Acombinationof appropriate to WorkStyles’ long-term benefits that
the inconclusiveness of the quantitative results and actuallyallowedtheprogramtoincreasethenumber
theextremelypositiveexperienceofworkingwitha ofitsofferings.Atthetimethatthismanuscriptwas
committed lead community partner (as well as wit- beingwritten,WorkStyleshadbegunofferingclass-
nessing and appreciating the exceptional nature of es twice per month instead of once.Therefore, as a
the WorkStyles program) led the lead quantitative result of presenting the long-term benefits of
doctoralstudent,inherroleasprojectmanagerofthe WorkStylestoo,theoverallfundingfortheprogram
study,tomakethedecisiontoincludetheentirequal- increased enough to allow for almost double the
itative piece in the final report to the funder. numberofannualprograms,whichextendsthereach
41
Pumaetal.
Table1
PromisingPracticesinThisCBPERProjectbyProjectDomain
ProjectDomain WhatWorked
University-CommunityPartnership •Sustainablesupportsystems
•Collaborationandcleardivisionoflabor
•Projectteamnurturedtheuniversity-communityrelationship
EvaluationActivities •Allpartnersinvolvedineveryphaseoftheevaluation
•Realisticgoalsandprojectexpectationsset
•Qualitativeandquantitativereportscombinedintoamore
comprehensivefinalreport
•Findingsverifiedwithprogramstaffmembersbeforedissemination
ofamoreformalreport
•Finalmeetingconvenedwithstakeholderstodiscusstheevaluation
findingsandtheiruse
ProjectOutcomes/Impact •Findingswereusedbythefunderandcommunitypartner
•Allpartnersbenefitedfromtheproject
oftheprograminmeetingthepre-employmentneeds other,aswellasacademicandwork-relatedsupport
ofmanymorerefugees. from the university professor and the director of
refugeeservices.Equallyimportantwasthetrusting
Promising Practices inThis CBPER Project
relationship between the refugees and the project
team, facilitated by active participation in the data
Everyresearchorevaluationprojectundertakenin
gatheringprocessbyWorkStylesstaffmemberswho
collaboration with community representatives
themselveswererefugees.Therefugeesprovidedthe
involvesdifferentparameters(e.g.,variedgoals;dif-
information without which there would not have
feringlevelsofinvolvement,expertise,andcommit-
been a qualitative study, and such trust is hard to
mentfromthevariousstakeholders;andawiderange
develop,andeasytoshatter.
of financial resources). As such, there is no single
Additionally,whatwasuniqueinthisparticularpro-
recipe for success for such a project. However, the
jectandwhichundoubtedlycontributedtoitsgeneral
collaborationamongthedifferentstakeholdersinthis
success,wasthehierarchyofresponsibilitiesassigned
casestudyresultedinseveralingredientswhichother
tograduatestudentsontheproject.Thequantitatively-
practitioners may find helpful when planning and
orienteddoctoralstudentwasapproachingtheendof
conducting their own projects in the community.
hercourseworkwhenthisevaluationstudytookplace.
ThesearelistedinTable1.Theyareorganizedinto
Assuch,shewastheprojectmanagerandoversawthe
three project domains: university-community part-
mergingofthequantitativeandqualitativeportionsof
nership, evaluation activities, and project out-
thestudyandwasultimatelyresponsibleforproducing
comes/impacts.
the final report. The qualitatively-oriented doctoral
University-CommunityPartnership studentwasinthemiddleofhercourseworkwhenthe
project was underway. Thus, while she had a lower
Theuniversity-communitypartnershipswasoneof
levelofoverallresponsibility,herresponsibilitieswere
the stronger aspects of this project, despite the fact
significant.Shewasinchargeofoverseeingtheentire
that the current refugee program participants were
qualitative portion of the study, which included,
not represented among the partners (a shortcoming among other things, training the first-year graduate
discussed later). There were many reasons that the studentsfromtheCBRclass.Thesegraduatestudents
university-community partnership worked: sustain- hadtheleastresponsibility,yettheystillcontributedto
able support systems between and among the part- the project by taking field notes, helping write the
ners; a sense of collaboration between the partners, focusgroupprotocolsand,insomecases,helpingcon-
but also a clear division of labor for participants on duct the focus groups and interviews.This hierarchy
theprojectteam;andtheuniversity-communityrela- waseffectivebecauseitdesignatedappropriaterolesin
tionshipwasnurtured. the research/evaluation process to the graduate stu-
Sustainable support systems. Due to the relation- dentsbasedontheirlevelofgraduateschooltraining.
shipoftrustthathadbeenbuiltbetweenthepartners, Despite this “built-in” hierarchy, the graduate stu-
everymemberoftheresearchteamfeltheorshehad dents’voiceswereequallyheardduringallphasesof
thesupportoftheothermembers.Thiswasespecial- theproject,whichisconsistentwithCBRprinciples.
lytrueforthedoctoralstudentsandtheWorkStyles Collaboration and clear division of labor.
manager, who received social support from each Collaboration,specificallythecreationofequalpartner-
42
ACaseStudyofaCommunity-BasedParticipatoryEvaluationResearch(CBPER)Project
ships,isoneofthethreeguidingprinciplesofCBRas WorkStyles manager attended an introduction to
well as CBPER which distinguishes them from tradi- research class that the university professor taught, to
tionalresearch(Rebacketal.,2002;Strandetal.,2003). talk about this particular project. These types of
Thisprojectwasdrivenbyapartnershipstructurethat actions extended above and beyond the responsibili-
recognized the strengths and expertise of WorkStyles ties/rolessetforthintheproject,andhelpedtonurture
staffanddoctoralstudents.Byhisownadmission,the the relationship between the lead community partner
WorkStylesmanagerhadlimitedresearchknowledge; and the university partner. The WorkStyles manager
however,hebroughtfirst-handknowledgeandpractical commented, “At the level that we were all involved,
expertisegainedthroughhisrelationshipwiththepro- the relationships developed more solidly, more real;
gram, with the refugees that it served, and with staff theinteractionsweresomuchmoregenuine.”
members who were also themselves members of the
EvaluationActivities
refugee community. This afforded him the ability to
understandtheresearchneededtobenefitWorkStyles. What worked best with regards to the evaluation
And while the doctoral students possessed technical itself were the “book end” activities—the planning
expertise and skills in quantitative, qualitative, and and dissemination/utilization of the findings.
mixed research design, data analysis, and knowledge Specifically, the funder, WorkStyles manager, and
dissemination,theprojectrequiredthemtomoveoutof universityresearcherswereallinvolvedintheplan-
familiarsocialandprofessionalnetworkstoworkwith ningprocess;thegoal/expectationfortheprojectwas
anew,diversegroupofpeopleandapproachestosocial reasonable;thefindingswereverifiedwithprogram
change.Becauseeachpartnerrecognizedeachother’s staffmembersbeforedisseminationofamoreformal
strengths and limitations and heard and valued their report;andameetingwithalloftheinvolvedstake-
expertise,therewasequalparticipationintheresearch holders was held to discuss the evaluation findings
process.Adivisionoflaborresultedwitheachperson andhowtheywouldbeused.
havingoneormoredesignatedtaskswiththeexpecta- Allrecognizedpartnerswereinvolvedinplanning
tionsofeachpartyspecifiedclearly.Asoneofthedoc- process. The funder, the WorkStyles manager, and
toralstudentsputit,“Everyonepulledtheirownweight; theuniversityrepresentativeswereallinvolvedinthe
no one was a weak link. Everyone was excited. planning process. The identification of the research
Everyonecametomeetingswithwhattheyneededto question—“Is WorkStyles having a short-term
comewith…noonedidn’tshowupornotdowhatthey impact on the employment outcomes of its gradu-
were supposed to do.” Thus, the academics did not ates?”—came from the funder, but because the
dominate;rather,reciprocitywasdemonstratedbythe WorkStyles manager was also involved in the plan-
lead community partner having choice in the level of ning,hewasabletovoicewhatheandhisfellowpro-
involvementintheprojectandbeingfullyengagedin gramstaffmembershopedtolearnfromthisCBPER
thecreationandcritiqueoftheknowledgecreated. project. As such, a multiple-outcome, mixed-meth-
Nurturingtheuniversity-communityrelationship.In ods project was designed to meet the needs of the
thisparticularcase,notmuchnurturingatthebegin- funderandleadcommunitypartner.
ningoftherelationshipwasnecessarybecauseasone Realisticgoalsandclearexpectations.Despitethe
of the lead graduate students put it, “We all really factthatthisprojecthadmultipleoutcomes,thepro-
clicked;everyone’spersonalityjustjivedfromthevery ject had a realistic goal and clear expectations.
beginning.”However,evenrelationshipsthatareeasy Because CBR projects strive for social change, stu-
at the beginning may eventually take some effort. In dents and faculty members might have lofty, ambi-
this particular case, the lead community partner and tious goals related to ameliorating the social ills of
the university partner were in constant contact via our society. To compound the problem, projects
email.Additionally, each partner was physically pre- involving university partners often have timelines
sentatthesiteoftheother.Forinstance,allfiveofthe definedbythelengthoftimeofauniversityquarter
doctoralstudentswhoworkedonthisprojectattended or semester, making accomplishing such elevated
one of the closing ceremonies (the ceremony, goals virtually impossible. This project had a nar-
described at the beginning of this article, that cele- rowly defined, straightforward goal: to answer the
bratesthecompletionoftheWorkStylesprogram,as question put forth by the funder. In this sense, our
wellastheculturefromwhicheachrefugeeparticipant project was an example of a small-scale effort at
comes). The WorkStyles manager recounts, “At one socialchange(Maguire,1993).
closingceremonyKate[oneofthegraduatestudents Combined quantitative and qualitative findings in
fromtheCBRclass]madeaquiltandpresenteditto finalreport.Onthebackendoftheproject,adecision
thestudents.Shealsobroughtinrepresentationsofher wasmadetocombinethequalitativeandquantitative
culturebecauseitwasValentinesDaysoshebrought reports to create a more comprehensive report.
in Valentines for all the students.” Additionally, the Combiningthefindingsgaveamuchmorecomplete
43