Motivating participation and improving quality of contribution in ubiquitous crowdsourcing

15 Pages·2015·1.52 MB·English

by  Jorge Goncalves

ComputerNetworks90(2015)34–48
ContentslistsavailableatScienceDirect
Computer Networks
journalhomepage:www.elsevier.com/locate/comnet
Motivating participation and improving quality of contribution
in ubiquitous crowdsourcing
JorgeGoncalvesa,∗,SimoHosioa,JakobRogstadiusb,EvangelosKarapanosb,
VassilisKostakosa
aDepartmentofComputerScienceandEngineering,UniversityofOulu,Penttikaiterankatu1,P.O.Box4500,FI-90014Oulu,Finland
bMadeiraInteractiveTechnologiesInstitute,UniversityofMadeira,PoloCientïficoeTecnológicodaMadeira,2ndFloor,9020-105Funchal,
Portugal
a r t i c l e i n f o a b s t r a c t
Articlehistory: Ubiquitouscrowdsourcing,orthecrowdsourcingoftasksinsettingsbeyondthedesktop,is
Received15October2014 attractinginterestduetotheincreasingmaturityofmobileandubiquitoustechnology,such
Revised2June2015 assmartphonesandpublicdisplays.Inthispaperweattempttoaddressafundamentalchal-
Accepted3July2015
lengeinubiquitouscrowdsourcing:ifpeoplecancontributetocrowdsourcinganytimeand
Availableonline21July2015
anyplace, why would they choose to do so? We highlight the role of motivation in ubiq-
uitouscrowdsourcing,anditseffectonparticipationandperformance.Throughaseriesof
Keywords:
UbiquitousCrowdsourcing fieldstudiesweempiricallyvalidatevariousmotivationalapproachesinthecontextofubiq-
Motivation uitouscrowdsourcing,andassessthecomparableadvantagesofubiquitoustechnologies’af-
Participation fordances.Weshowthatthroughmotivationubiquitouscrowdsourcingbecomescomparable
Performance toonlinecrowdsourcingintermsofparticipationandtaskperformance,andthatthrough
Engagement motivationwecanelicitbetterqualitycontributionsandincreasedparticipationfromwork-
ers.Wealsoshowthatubiquitoustechnologies’contextualcapabilitiescanincreasepartici-
pationthroughincreasingworkers’intrinsicmotivation,andthatthein-situnatureofubiqui-
toustechnologiescanincreasebothparticipationandengagementofworkers.Combined,our
findingsprovideempiricallyvalidatedrecommendationsonthedesignandimplementation
ofubiquitouscrowdsourcing.
©2015ElsevierB.V.Allrightsreserved.
1. Introduction raisesnewpossibilities,italsoraisesanimportantquestion
ofmotivation:ifpeoplecancontributetocrowdsourcingany-
Crowdsourcing work and the associated distribution of timeandanyplace,whywouldtheychoosetodoso?
micro-tasksacrosslargenumbersofindividualsarebecom- Whiletheissueofmotivationhasbeenalong-standing
ingincreasinglypopularinsettingsbeyondthedesktop,thus concerninthedesignofcomputersystemsandonlineser-
enablingawiderangeofapplications.Ubiquitoustechnolo- vices, new technologies require that new motivational ap-
gies,suchassmartphonesandpublicdisplays,arenowma- proaches are developed, adapted, and validated. In terms of
ture enough to allow users to contribute to crowdsourcing crowdsourcing, research in psychology, sociology, manage-
taskswhereverandwhenever.Whiletheincreasedeasewith mentandmarketingprovideasolidtheoreticalbasisonhu-
whichitisnowpossibletoparticipateincrowdsourcingwork manmotivation[36].However,thesetheoreticalapproaches
typically have to be adapted and fine-tuned for a crowd-
sourcingsetting.Atthesame,bymotivatingworkerstocon-
tribute more, task requesters can unwillingly make them
∗ Correspondingauthor.Tel.:[phone].
moresusceptibletoqualitycontrolissues[36]socarefulmo-
E-mailaddress:[email protected].fi,[email protected].fi
(J.Goncalves). tivationalconsiderationshavetobetakenintoaccount.
http://dx.doi.org/10.1016/j.comnet.2015.07.002
1389-1286/©2015ElsevierB.V.Allrightsreserved.
J.Goncalvesetal./ComputerNetworks90(2015)34–48 35
In addition to accounting for human behaviour, a moti- totheworkers,anywhereandanytime.Mostoftheseplat-
vational approach also needs to account for the technolo- formshavebeendeployedindevelopingcountriestargeting
gies and context of use. Ubiquitous technologies are par- low-incomeworkersprovidingthemwithsimpletasks,e.g.
ticularly challenging as they are typically in the hands of [15,25]. Recent advances in mobile technologies have also
users,awayfromthecontrolsofalabsetting,andnormally allowed for more intricate and creative tasks. For instance,
lackanyidentificationmechanisms.Therefore,crowdsourc- Wallah,amobilecrowdsourcingplatformforAndroidOSim-
ingusingthesetechnologiesmayproduce“noisy”resultsdue plementscachingforofflinesituationsandaimstominimize
tounpredictablebehaviourormisappropriationfromusers the impact of different screen sizes of smart phones [40].
[28,59].Thus,priorworkfurtheremphasizestheimportance Morebroadly,thelocation-baseddistributionofcrowdsourc-
of appropriate motivational approaches to address these ingtaskshasalloweditsworkerstoperformreal-worldtasks
challenges. inapeer-to-peerfashion.Someexamplesofthisincludepro-
Inthispaperweadapt,presentandvalidatemotivational vidinglocation-awarerecommendationsforrestaurants[1],
approachesforubiquitouscrowdsourcingbydrawingonex- providinginstantweatherreports[1],orauthoringnewsar-
tensiveliteratureandleveragingourownresultsfromfour ticlesbyrequestingphotographsorvideosofcertainevents
casestudies.Theseapproachesdrawonpriorliteratureinhu- fromworkers[63].
manbehaviour,accountforubiquitoustechnologies,andare Recently,anothercommunityhasdevelopedaroundthe
validatedinfieldtrialstoestablishtheireffectonparticipation topicofcrowdsourcingmeasurementsandsensing.Thispar-
andqualityofcontribution.Weanswerfourimportantques- ticipatory sensing movement is also referred to as “Citizen
tionsonthemotivationalaspectsofubiquitouscrowdsourc- Science”[49]andreliesonmobilizinglargepartsofthepop-
ingregardingthepossibilityofelicitingaltruisticcontribu- ulationtocontributetoscientificchallengesviacrowdsourc-
tions,theeffectivenessofvariousmotivationalapproaches, ing.Oftenthisinvolvestheuseofsmartphonesforcollecting
thepotentialadvantagethatubiquitoustechnologiescanof- data[6]orevendonatingcomputationalresourceswhilethe
ferforcrowdsourcing,andfinallyonthesituatednatureof phoneisidle[2].
ubiquitouscrowdsourcing: Despite the appeal of mobile phones, using them for
crowdsourcing requires workers’ implicit deployment, con-
1. Canubiquitouscrowdsourcingworkaltruistically?Wean-
figuration and utilising users’ own hardware. For example,
swerthisthroughacasestudywhereweexperimentally
inSMS-basedcrowdsourcing,participantsneedtoexplicitly
compareresultsobtainedbetweenubiquitousandonline
signupfortheservice,atthecostofatextmessageexchange.
crowdsourcing. Specifically, we establish a baseline as-
Thischallengesrecruitmentofworkers,asanumberofsteps
sessment showing that performance in ubiquitous set-
needtobeperformedbeforeaworkercanactuallystartcon-
tingswithoutpaymentiscomparabletoonlinesettings
tributingusingtheirdevice.Alternatively,apassiveapproach
withpayment,andworthyoffurtherinvestigation.
of crowdsourcing tasks to workers is to embed public dis-
2. Can psychological empowerment motivate ubiquitous
playsintoaphysicalspaceandleveragingworkers’serendip-
crowdsourcing? We answer this by validating psycho-
itous availability. Crowdsourcing using public displays re-
logical empowerment approaches as motivators for
quireslittleeffortfromtheworkertocontribute[19,24],low-
participation in ubiquitous crowdsourcing. Specifically,
eringthebarrierstocontributionfromaworkers’perspective
wevalidatehow3typesofpsychologicalempowerment
byminimisingtheinitialeffort.Furthermore,itallowsfora
affect participation and contribution in crowdsourcing
geofencedandmorecontextuallycontrolledcrowdsourcing
onmobilephones.
environment[24],thusenablingtargetingcertainindividuals
3. Cancontextualcuesmotivateubiquitouscrowdsourcing?To
[19,20],leveragingpeople’slocalknowledge[21,28]orsimply
answerthiswetesttheeffectofpresentingcontextualin-
reachinganuntappedsourceofpotentialworkers[27,29].
formationonengagement.Specifically,weevaluatehow
A reflection on the effective facilitation mechanism for
thepresentationoflocationcuesaffectsparticipationina
public displays to motivate users to deliver reliable and
“crowd-mapping”setting.
meaningful feedback is lacking but also imperative. Most
4. Can situatedness motivate ubiquitous crowdsourcing?
prior research has reported the use of public displays
Specifically,weinvestigatehowthesituatednessofubiq-
for hedonic services (e.g., games, opinion disclosure) or
uitoustechnology canmotivatepeopletoparticipatein
information-basedservices(e.g.,informationboards)thatof-
in-situfeedbackcollection.
ferinstantbenefitstousers[5,38].Thereisalackofdelibera-
Weconcludethepaperwiththelessonslearnedthrough- tiononthepossibilityofusingpublicdisplaysinanaltruistic
out our case studies, discussion of other potential mo- manner,suchasfornon-paidcrowdsourcing.Pragmatically,
tivational approaches in ubiquitous crowdsourcing and asuccessfuldemonstrationofthepotentialofpublicdisplays
empirically validated recommendations on the design and foraltruisticservicesimpliesapossiblefuturedirectionfor
implementationofubiquitouscrowdsourcing. publicdisplaysresearchandpractice.
Despitethevariousbenefitsofpublicdisplaysforcrowd-
2. Relatedwork sourcing, there are some serious drawbacks. For instance,
thewalk-up-and-usenatureofpublicdisplayscanresultin
2.1. Technologicalopportunitiesforubiquitouscrowdsourcing limitedusabilityandaccessibilityoftasks,withlessrichin-
terface controls than a standard desktop environment or a
Crowdsourcing with ubiquitous technologies is increas- mobile phone. This means that not all types of tasks can
ingly gaining researchers’ attention [64,65], especially on be crowdsourced on a public display. Another drawback is
mobile phones. This has allowed researchers to push tasks thatthemaintenanceofpublicdisplaysismoredifficultthan
36 J.Goncalvesetal./ComputerNetworks90(2015)34–48
maintaining an online server and can incur higher initial dlerandKapelner[8]varymeaningfulnessofcontext(intrin-
costs.Thescalabilityofcrowdsourcingusingpublicdisplays sic)inafixeddiminishingfinancialrewardstructure.Finally,
isdirectlydependentonthenumberofdisplaysallocatedto thetaskusedinChandlerandKapelner[8]resultedinvery
thesystemandtheiraudiences.Evenso,displaysarebecom- highperformancelevels,suggestingapossibleceilingeffect
ingincreasinglyaffordable,andresearchershavesystemati- ontheinfluenceofintrinsicmotivation.
callyattemptedtoidentifynovelapplicationsforthistech-
nology.Also,researchoninteractivedisplaysinpublicspaces 2.3. Adaptedmotivationapproachesforubiquitous
hasoftennotedthatuserstypicallydemonstrateplayfuland crowdsourcing
exploratorybehaviourwhenusingthistechnology.Asare-
sult,thecollectionoffeedbackonsuchdisplaysisdifficult, Herewedescribethemotivationalapproachesthatwere
andresultstendtoberathernoisy[28]. adaptedinourcasestudies.Thesewerechosenbasedonpre-
viousworkdemonstratingtheirsuccessinmotivatingpartic-
2.2. Motivationinonlinecrowdsourcingmarkets ipationaswellasimprovingcontributionquality.
Onewaytoachievehigherlevelsofmotivationinubiqui-
Whydopeopleparticipateincrowdsourcingmarkets,and touscrowdsourcingisthroughpsychologicalempowerment.
what predicts their performance? A traditional “rational” Its role in improving citizen participation has been high-
economic approach to eliciting higher quality work would lighted in the past [68]. Psychological empowerment inte-
betoincreaseextrinsicmotivation,e.g.,anemployercanin- gratesperceptionsofpersonalcontrol,aproactiveapproach
creasehowmuchtheypayforthecompletionofatask[17]. tolifeandacriticalunderstandingofthesocio-politicalen-
Someevidencefromtraditionallabormarketssupportsthis vironment[69].Generallyspeaking,commitmentinachiev-
view:Lazear[41]foundworkerstobemoreproductivewhen ingpersonallyrelevantaimsstrengthensagency[7]andcon-
theyswitchedfrombeingpaidbytimetobeingpaidbypiece. sequentiallygeneratespsychologicalempowermentthrough
AnexperimentbyDeci[14]founda“crowdingout”effect theseactions.Someofthefactorsofpsychologicalempow-
ofexternalmotivationsuchthatstudentspaidtoplaywith ermentthathavebeenrepetitivelyfoundtomotivatecitizen
apuzzlelaterplayedwithitlessandreportedlessinterest participationthatarenormallyusedincollectivegoodset-
than those who were not paid to do so. In the workplace, tingsare:
performance-basedrewardscanbe“alienating”and“dehu-
• perceivedself-efficacy:thedegreetowhichindividualsbe-
manizing”[16].Iftherewardisnotsubstantial,theperfor-
lieve they have the capabilities to achieve the desired
manceislikelytobeworsethanwhennorewardisofferedat
goals[46],
all;insufficientmonetaryrewardscanactasasmallextrinsic
• senseofcommunity:therelationshipbetweentheindivid-
motivationthattendstooverridethepossiblylargereffectof
ualandthesocialstructure[52],and
thetask’slikelyintrinsicmotivation[18].Giventhatcrowd-
• causalimportance:anindividual’sbeliefsabouttherela-
sourcingmarketssuchasMechanicalTurktendtopayvery
tionshipbetweenactionsandoutcomes.[50].
littlemoneyandinvolverelativelylowwages[48],external
motivationssuchasincreasedpaymayhavelesseffectthan Another way to motivate users’ participation in ubiqui-
requestersmaydesire.Indeed,researchexaminingthelink touscrowdsourcingisthroughtheuseofcontextualizedin-
betweenfinancialincentivesandperformanceinMechanical formation. Contextualizing information has been shown to
Turkhasgenerallyfoundalackofincreasedqualityinworker helptapintoindividual’sepisodicmemories[61]byallow-
output[42].Therelationshipbetweenpriceandqualityhas ingmentally“re-living”specificlifeexperiencesandimprove
alsohadconflictingresultsinothercrowdsourcingapplica- recollection by thinking back in detail to past personal ex-
tions such as answer markets (e.g., [26]). Although paying periences [60]. Reflection can be used to examine patterns
morecangetworkdonefaster,ithasnotbeenshowntoget ofpastexperiences,whichmayprovideusefulinformation
workdonebetter. about general level of physical activity or emotional states
Anotherapproachtogettingworkdonebettercouldbe indifferentsituations,allowingthepersontorelatetoother
increasing the intrinsic motivation of the task. Under this data [60]. Furthermore, literature suggests the use of digi-
view, if workers find the task more engaging, interesting, tal cues as appropriate contextual cues because our mem-
or worth doing in its own right, they may produce higher oryisareconstructiveprocessmediatedbytriggersfromev-
quality results. Unfortunately, evidence so far has not fully erydayevents[4,11].Themostusedtypesofdigitalcuesare
supportedthishypothesis.Forexample,whilecrowdsourc- visualcues[39,66]orlocationcues[67],whichcantrigger
ingtasksframedinameaningfulcontextmotivateindividu- everydayrecall,promoteattentivenessandleadtoincreased
alstodomore,theyarenomoreaccurate[8].Ontheother participation.
hand,workbyRogstadiusetal.[56]suggeststhatintrinsic Anotherapproachtoleveragingubiquitouscrowdsourc-
motivationhasasignificanteffectonworkers’performance. ingisthroughsupportingopportunisticparticipation.Prior
Onehastonotehoweveranumberofquestionsandmethod- workhasshownthatallowingpassersbytounderstandsitu-
ologicalquestionsthatareyettobesettled.First,priorstud- atedandcontextuallyrelevantinformationcanleadtogen-
ieshavemethodologicalproblemswithself-selection,since uinely insightful contribution [3]. For instance, public dis-
workers may see equivalent tasks with different base pay- plays have become a viable medium for such opportunistic
mentorbonusesbeingpostedeitherinparallelorserially. contribution. Supporting this, De Cindio et al. [9] observed
Second,veryfewstudiesbesides[56]havelookedatthein- that people leave feedback often during so called peak or
teractionbetweenintrinsicandextrinsicmotivations;Mason protest moments, when the circumstances for public dis-
&Watts[42]varyfinancialreward(extrinsic),whileChan- courseordisapprovalareright.Theseresultstogetherraise
J.Goncalvesetal./ComputerNetworks90(2015)34–48 37
Fig.1. Taskimage(varyingcomplexity).
the question of whether situated feedback mediums could pants.Furthermore,weinvestigatedwhateffectmotivation
beleveragedtoreachpeopleduringthesekeymomentsfor canhaveinsuchasetting.
discourse. The experiment consisted of testing a crowdsourcing
Onemayexpectthesemomentstooccurwhencitizens service on public displays and on Mechanical Turk. The
confrontapublicdisplayinacityandaregiventhepossibil- taskusedinourexperimentisthecountingtaskdescribed
itytoleaveinstantfeedbackaboutalocallyremarkableand by Rogstadius et al. [56] in which workers are asked to
topicalissuethatinvadestheirterritory.Publicdisplaysalso countmalaria-infectedbloodcellsonimagesofapetridish
foster sociality and group use by nature [37], and eliciting generatedalgorithmically(Fig.1).Wedeployedthistaskon
contributionfromgroupsofusersisofteneasierthanfrom bothMechanicalTurk,aswellasasetofpublicinteractive
individuals[28].Further,thewell-knownhoneypoteffect[5] displays deployed throughout our university campus. Be-
can be leveraged to our advantage in spreading awareness causethetaskwasidenticalacrossthesetwodeployments,
aboutthefeedbackchannelamongnearbypotentialusers. we were able to make a direct comparison between the
Archetypalfeedbackapplicationsonpublicdisplaysuti- two.Nomoneywasgiventotheparticipantswhousedthe
lize typing in some form as their main input modality. publicdisplays,whileparticipantsonMechanicalTurkwere
Brignull and Rogers reported Opinionizer [5], a system that rewardedwith0,3or10centspertaskcompleted.
combinedaprojectedscreenwithalaptoptotypefeedback
andconverseabouttheeverydaycontextsitwasdeployedin. 3.1.1. Motivationalapproach
Theyintroducedthehoneypoteffectandemphasizedsocial Focusingonourpublicdisplaydeploymentwevalidated
pressureandawkwardnessthatusersoftenfeelwheninter- twomotivationalapproaches.Thefirstapproachwasbased
actingpublicly.Aplayfulfeedbackapplication,connectedto onintrinsicmotivationinwhichweidentifiedfromliterature
social networking services and utilizing a virtual keyboard twotypes:enjoyment-basedandcommunity-based[34]. We
and a web camera for feedback was introduced by Hosio usedoneconstructpermotivator,toenablereliabletesting
et al. [30]. Studies with Ubinion also highlighted situated onapublicdisplay.
publicdisplaysbeingfitforacquiringcontextuallyrelevant
feedback. • Taskidentity:Aworkerperformsataskbecauseheknows
thathisworkwillbeused(e.g.,writingaproductdescrip-
tionforawebsite)[34].
• Community identity: A worker who only accepts tasks
3. Casestudies
fromrequesterswithagoodreputationbecausetheyare
knownasavaluablesupporterofthecommunity[34].
3.1. CaseStudy1:canubiquitouscrowdsourcingwork
altruistically?
Usingthesewederived4conditionsofmotivationforour
experiment.Thesewereconveyedthroughtheintroductory
In this case study we summarise our work on directly
textontheinstructionpage,whichwasmanipulatedbased
comparingcrowdsourcingconductedinanonlineandaubiq-
onthecondition:
uitoussetting.Specifically,wecontrastthecrowdsourcingof
anidenticaltaskthatwasconductedinMechanicalTurk,as • Control(nomotivation):“Weinviteyoutoidentifyblood
wellasonpublicinteractivedisplays.Thisworkhasbeenpre- cellsinfectedwithmalariaparasites.”
viouslyreportedin[19,56]andherewesummarisethepoints • Enjoyment-based(taskidentity):“Weinviteyoutoidentify
thataremostpertinenttoourdiscussion. blood cells infected with malariaparasites. This in turn
Thiscasestudywasthefirst(toourknowledge)attempt willhelpproducebettersoftwaretoimprovemalariacell
toinvestigatealtruisticuseofinteractivepublicdisplaysin detection.”
naturalusagesettingsasacrowdsourcingmechanism.Our • Community-based (community identity): “We invite you
goal was to explore the possibility of workers contributing to identify blood cells infected with malaria parasites.
onanon-personaldevicethushighlightingthe“ubiquitous” ThisinturnwillhelpOulumedicalscientistsontheirre-
natureofthistaskwhilealsonotactivelyrecruitingpartici- search.”
38 J.Goncalvesetal./ComputerNetworks90(2015)34–48
• Both enjoyment & community-based (task & community quickly the tasks were completed, and how much time it
identity):“Weinviteyoutoidentifybloodcellsinfected takes to have a large number of tasks completed. Overall,
withmalariaparasites.Thisinturnwillhelpproducebet- ouranalysisshowedthataccuracyresultsobtainedonMe-
tersoftwaretoimprovemalariacelldetectionandhelp chanicalTurkwereabout10%higherwhencomparedpub-
Oulumedicalscientistsontheirresearch.” licdisplay.However,participants’performancewassignifi-
cantlyhigher(p<.01)inconditionswhereamotivatorand
The second motivational approach we tested was fact-
thefact-checkingquestionwerepresentwhencomparedto
check,withtwolevels:presentandabsent.Fact-checksim-
MechanicalTurk.Inaddition,workersonthepublicdisplay
plyconsistsofaskingusersaquestionwhoseanswerisvery
weremorelikelyto“giveup”afteracertainpointofcom-
simpletoanswer.Thiscanbeaneffectivewaytofilterout
plexityhighlightingtheneedforcarefultaskassignmentin
non-seriousanswersandimprovetheoverallqualityofthe
thismedium.Finally,therateofuptakeoftasksonthepublic
answersgiven[35].Animportantcharacteristicofthisstepis
displaywasmuchhigherthanonMechanicalTurkreaching
thatthequestionmustbeeasytoanswer,anditmustbeclear
1200taskscompletedin25dayscomparedtothenon-paid
totherespondentthattheexperimentersalsoknowthean-
versiononMechanicalTurkthatonlyreached100taskscom-
swer.Inthoseconditionswherethefact-checkwaspresent,
pletedinover45days.
we showed users a question just before starting their first
task.Thequestionwas:“Whatisthenameofourplanet?”
3.1.3. Emergingbehavioursandtheirimpactonperformance
Userswouldthenhavetoselecttheiranswerout4possibil-
Understandingthesocialdynamicsaroundsituatedtech-
ities(Earth,Saturn,Mars,andJupiter)orderedrandomlyto
nologiesiscrucialfortheimprovementofcrowdsourcingin
avoidbias.
these mediums. By understanding the potential emergent
workerbehavioursinubiquitouscrowdsourcing,taskscanbe
3.1.2. Performancecomparisonbetweenbothdeployments
designedtocaterforthemoreproductiveworkerarchetypes.
OnMechanicalTurk,atotalof158workerscompletedat
Inonlineandmobilecrowdsourcingstudieswherepersonal
leastonetaskwhileonthepublicdisplaystherewere482
devicesareused,workersarenormallytreatedasablackbox.
unique sessions in which at least one task was completed.
There are only a few exceptions in the literature [57,58] in
Ouranalysisshowedthatwhenitcomestocrowdsourcing
which task requesters can infer worker performance from
using public displays, intrinsic motivators, but not fact-
the way they conduct the tasks. Ubiquitous crowdsourcing
check,hadasignificanteffectonaccuracy,completiontime
offersthepossibilitytodirectlyobservepeoplecompleting
andtaskscompleted.Workersthatwerepresentedwithany
crowdsourcing tasks. Similar but simpler approaches pro-
ofthe3psychologicalconstructs(i.e.excludingthecontrol
viderequestersmorevisibilityintoworkerbehaviour,such
condition)weremoreaccurate(F(3,1192)=13.93,p<.01),
asoDesk’sWorkerDiary,whichperiodicallytakessnapshots
spentmoretimedoingeachtask(F(3,1192)=6.07,p<.01)
ofworkers’computerscreens.
and completed more tasks per session (F(3,474) = 3.45,
Here,wehadtheuniqueopportunitytoobservepartic-
p=.02),thuspayingoverallmoreeffortinperformingthe
ipants’attitudesandsocialcontextwhencompletingtasks.
tasks. While previous work has found that unusually long
Weusedvideorecordingtocaptureallinteractionswithone
taskcompletiontimeindicatesdistractionandpoorcommit-
of the public displays during the study (Fig. 2). Our video
ment[54],inthiscasethoseinthecontrolconditionsended
observationsincluded123instancesofinteraction.Ourcon-
upspendingsignificantlylesstimeperformingthetasks,par-
tent analysis coded these videos concurrently by a group
ticularlythemorecomplexones.Furthermore,thepresence
ofresearchersusingopenandaxialcoding,thusidentifying
ofthefact-checkalsoactedasanimportantqualitycontrol
emergingthemesofbehaviour.Asreportedin[19],thisanal-
mechanism. Those that answered this question correctly
ysisconfirmedinstancesofthebehavioursthatweinitially
were far more likely to perform the following tasks more
notedinourin-situobservations,butalsorevealedseveral
accuratelyandingreaternumber,whilenothavingasignif-
new behavioursthat peopleexhibited when using the dis-
icantinfluenceinthetimetheywouldbewillingtospend.
play.Thesixidentifiedbehaviourswere:
Furthermore,wecomparedourtwodatasets(Publicdis-
play vs. Mechanical Turk) in terms of: accuracy and rate • Ignorer: passers-by that ignored the display, exhibiting
of uptake of tasks. The rate of uptake is indicative of how whatisoftenreferredasdisplayblindness[44],and
Fig.2. Thedifferenttypesofbehaviourfrequentlyobservedaroundthedisplay.Ignorer:Thisisthemosttypicalscenariowhereapasserbycompletelyignores
thedisplay.Attractor:person“A”startsusingthescreen,person“B”becomesattractedandapproaches,andeventually“B”leaveswhile“A”remainsonthe
display.Herder:“A”approachesandusesthedisplaywhileagroupobserveshim.Loner:“A”approachesandusesthescreenforarelativelylongperiodoftime,
whilepassersbyignorehim.Repeler:“A”startsusingthescreenwhile“B”usesbodylanguagetoapplysocialpressureto“A”toleave.Unlocker:“A”briefly
interactswiththedisplaywithoutstoppinghiswalk.
J.Goncalvesetal./ComputerNetworks90(2015)34–48 39
• Unlocker: those that actually unlocked the screen but mechanism to identify non-serious respondents. Finally,
completednotasks.Theseaccountforthehighnumber we found that crowdsourcing on public displays without
ofcuriosityclicksmentionedpreviously. explicit recruitment can produce comparable performance
• Herder: individuals would approach the display with a even to paid studies on Amazon’s Mechanical Turk. This
groupofpeople,completesometasksandthenleavewith highlightsthedifferenceinthecomparedmediums,which
thegroup.Theothermemberswouldadoptapassivepo- wasexpectedsincetheytapintoverydifferentpopulations.
sition behind the herder, in a way that suggested they Onlineworkerstypicallyexpecttogetpaidfortheirwork.By
werenotapplyingsocialpressurebutratherobserving, completingnon-paidtaskstheyarepracticallylosingmoney
• Loner: individuals that approached the display alone bynotusingthetimetocompletehigher-payingtasks.Thus,
and typically spent more time than others completing work motivated by altruism is not particularly suitable for
tasks. labormarketssuchasMTurk.Ontheotherhand,ubiquitous
• Attractor: attracted others to join them on the display, crowdsourcing offers a setting that is rather suitable for
commonlyreferredasthehoneypoteffect[5],andcom- enticing volunteers and eliciting altruism in the context of
pletetasksjointly. crowdsourcing.
• Repeller:appliedsocialpressuretotrytomaketheworker
leave the display. Instances of repellers also happened 3.2. CaseStudy2:canpsychologicalempowermentmotivate
whengroupsoftwoormorepeopleapproachedthedis- ubiquitouscrowdsourcing?
play.
An important limitation of the previous case study was
Therelativefrequencyofthebehaviourpatternsthatac- thatduetoits“in-the-wild”naturewewerenotabletoiden-
tivelyinteractedwiththedisplaywas:Loner19%,Attractor tifyandkeeptrackofallpeoplewhotookpartintheubiq-
11%,Herder6%,Repeller14%,Unlocker44%.Theremaining uitouscrowdsourcingpart,andthereforeitisimpossibleto
6%ofinteractionsdidnotfitthedescriptionoftheaforemen- reliablyassessindividuals’performance.Toaddressthislim-
tioned behaviour patterns. Loners completed on average a itationinattributingperformanceandfindingstoindividual
higher amount of tasks (M = 4.91, SD = 1.04), followedby participants,weconductedastudywhereparticipantswere
attractors (M = 3.71, SD = 4.53), herders (M = 3.43, SD = recruited.Herewewereinterestedincollectingrichdataon
1.29)andfinallyrepellers(M=1.29,SD=.59).AKruskal– theeffectofpsychologicalempowermentonindividualpar-
Wallistestshowedthattherewasasignificantdifferencein ticipants, and how that may affect their performance, be-
average number of tasks completed between the different haviour,andperceptions.
behaviours(χ2(4)=22.18,p<.01).Post-hocanalysisusing
theMann–Whitneytestsshowedthattherewereonlyasig- 3.2.1. Motivationalapproach
nificantdifferencebetweenlonersandrepellersintermsof Inthiscasestudyweevaluatedtheimpactofthreeprin-
averagenumberofcompletedtasks(U=26.04,p<.01).As ciplesofpsychologicalempowerment,namelyperceivedself-
foraccuracy,aKruskal–Wallistestshowedthattherewasno efficacy,senseofcommunityandcausalimportance,onpublic
significantdifferenceinaccuracybetweenthedifferentbe- transportpassengers’motivationtoreportissuesandcom-
haviours(χ2(4)=7.99,p=.09).Theseresultssuggestthat plaints while on the move. We chose SMS as the commu-
appealing to certain workers can improve the task uptake nicationmediumforrecruitmentandenablingparticipation
withouthavingasignificantimpactonaccuracy.Atthesame andinteractionwithbusriders.Formoredetailsonthisstudy
italsohighlightstheimportanceofmotivationwhencollect- pleasereferto[23].
ingcrowdwork. Whenparticipantssignedupforthestudy,theywerein-
The latter two behaviours (attractor and repeller) ulti- vited to report problems or make suggestions for the im-
matelyledtoadisturbanceanddelayinthecompletionof provementofthebusservice.TheywereinformedviaSMS
the tasks. In other words, this resulted in the opposite of thatthestudywasconductedandcontrolledbytheUniver-
peerpressureinthatworkersinsteadofbeingpressuredto sity, and their comments would eventually be shared with
dowell,theywouldengageinperformativeacts[27]result- thebuscompany.Theywerealsoinformedthattheycould
inginnon-seriouscompletionoftasks.Previousworkhasre- submit an SMS with ‘Help’ to receive further tips, and an
portedthatinsomecasestheengagementwiththeseinter- SMSwith‘Unregister’tooptoutofthestudy.Followingtheir
activepublicartefactsemergesonlywhentheoverallsocial registration,participantswererandomlyallocatedtooneof
contextprovidesa“licensetoplay”[32].Inthecaseofplayful thefourconditions:onecontrolconditionandthreerepre-
applicationsorgames,thisdoesnotmatterandcanevenact sentingthefactorsofpsychologicalempowermentthatwe
asacatalysttouse[37],butwhencollectingmeaningfuldata wantedtoaffect.
fromthepublic,itmaybebeneficialtoattractmoreloners At the end of each day (8 p.m.) participants received a
thangroups. singlemotivationalSMSreflectingtheconditiontheywere
allocated to. The participants in the Control condition also
received a message that did not involve psychological em-
3.1.4. Take-away
powermentbutsimplythankedthemfortheirparticipation.
Wefoundthatthroughthecontrolleduseofmotivational
SomeexamplesofmotivationalSMSsenttoparticipantsfor
design performance can be significantly improved, and
eachconditionare:
that community-based and enjoyment-based motivational
approaches can be successful. We also found that when • Perceived self-efficacy: “Your contributions have been
crowdsourcingusingpublicdisplays,fact-checkingisnotas great.Pleasecontinuecontributingwheneveryoufeelit
effective as in online settings, but it still acts as a reliable isnecessary.”
40 J.Goncalvesetal./ComputerNetworks90(2015)34–48
• Senseofcommunity:“Thanksforbeingpartofthismove- customers,e.g.“Thedriverwasdrivingtoofastwiththis
menttomakepublictransportationmoreenjoyabletoev- rain”,
eryone.” • Otherpassengers’behaviour(N=6),e.g.“Busfullofkids
• Causal importance: “Thank you for your comments. All constantlyshovingme”,
yourmessageswillbetakenintoaccount.” • Qualityofinfrastructure(N=46)suchasnon-operational
• Control:“Thankyouforparticipating.” vending machines or inappropriate bus stops and shel-
ters,e.g.“Theelectronicscheduleshaveconstanterrors,I
Wehypothesizedthatincreasedpsychologicalempower-
donottrustthemanymore”,
mentwillleadto: • Qualityofservice(N=85)referringtothecost,orthequal-
ityoftheoverallservice,e.g.“Neverenoughchangewhen
• Increased participation, which can be measured by the
Ibuyticketsonboard”and
numberofsubmittedreports[46,50,52].
• Suggestions (N = 92) either for the improvement of ex-
• Improvedqualityofcontribution,whichcanbeanalysed
isting services or development of new ones, e.g. “There
throughthesubmittedreports.
shouldbeabusonlyforstudents”.
Aswehavereportedpreviously[23],wetookmeasuresto
Overall, no significant relationship was found between
ensurethateverySMSwesenttoeachparticipantreflected
motivationandcategoryofreport.However,thesuggestions
asingleempowermentstrategy.Inadditionweensuredthat
categorywasthemostpopularforparticipantsintheSelf-
eachparticipantreceivedavarietyofmessages(ratherthan
efficacy and Causal importance conditions (30% and 32%
theexactsamemessageeveryday),andeachmessageshould
oftheirtotalreports,respectively),butnotforparticipants
only reflect the empowerment strategy of the participant’s
ofSense-of-communityandtheControlcondition(19%and
condition. To satisfy these requirements, we constructed a
18%).Whendistinguishingreportsbetweensuggestionsand
poolof14messagesforeachofthe3manipulatedconditions
complaints (i.e., the remaining five categories) we found a
(7messagesforactiveparticipantsand7messagesforpas-
significantrelationshipbetweenthemotivationalapproach
siveparticipants),plus2messagesforthecontrolcondition andthecategoryofreports(χ2=9.05,df=3,p=.03).
(1messageforactiveparticipantsand1messageforpassive
Wealsofoundasignificanteffectofmotivationontheto-
participants). We chose to have only two messages for the
talnumberofreportsparticipantssubmitted(F(3,61)=5.44,
controlconditionasitmodelscurrentsystemsanditinten-
p<.01).ParticipantsinthePerceivedself-efficacyandCausal
tionally does not change. This resulted in 44 distinct mes-
importance conditions contributed significantly more re-
sages.
portswhencomparedtothoseintheControlcondition(p<
To verify that the messages reflected the intended em-
.01 and p < .05, respectively), but this was not true for
powermentstrategy,werecruited10colleaguesandbriefed
thoseintheSenseofcommunitycondition(p>.05). Some
themoneachpsychologicalconstruct.Wethenaskedthem
participants mentioned that our feedback messages acted
tousecardsortingtoassigneachofthe44messagestoone
as a reminder while adding to motivation to provide more
type of psychological empowerment or to the control con-
feedback.
dition. Overall, 92.1% of the assignments were accurate. In
Finally,weobservednosignificanteffectofthemotiva-
addition, we interviewed five colleagues who had not par-
tionapproachonthelengthofthereportsintermsoftotal
ticipated in the card sorting, discussing how our messages
characters(F(3,357)=2.056,p=.11)orthetimeofdaythe
madethemfeelandwhattheirthoughtswereonthem.The
reportwassent(F(3,357)=1.24,p=.30).Reportswereover-
responses confirmed that nearly all messages instilled the
allspreadthroughoutthedaystartingrangingfrom7amto
feelings of psychological empowerment we intended. We
11pmwiththespikesoccurringduringrushhours(morning
usedthisfeedbacktofurtheriterateonthemessagesuntil
rushhour:8am-9am,eveningrushhour:7–8p.m.).
weweresatisfiedtheyreflectedtheintendedpsychological
empowerment.
3.2.3. Take-away
Our analysis showed that our manipulation of the mo-
3.2.2. Results
tivational approach had two significant effects. First, those
Intotalwehad65participantsandreceived354reports
participantswhoreceivedmotivationalmessagesweremore
fromparticipants.Ofallreports,109(30%)wereinthePer-
likelytoprovidesuggestionsratherthancomplaints.Second,
ceivedself-efficacycondition,88(25%)inSenseofcommu-
those participants provided more reports and participated
nity,94(27%)inCausalimportanceand63(18%)intheCon-
more frequently. Overall, our 65 participants revealed that
trol condition. All reports were subjected to a qualitative
Self-efficacyandCausalimportanceincreasedparticipation
content analysis [31]. This process consisted of open and
whilealsoimprovingthequalityofcontribution.
axialcoding,andwasconductedindependentlybytwore-
searchers.Theresultingcodingschemewasdiscussedandit-
3.3. CaseStudy3:cancontextualcluesmotivateubiquitous
erated,andallreportswereclassifiedinoneofsixcategories.
Interraterreliabilitywassatisfactory(Cohen’sK=0.85).The crowdsourcing?
sixcategorieswereidentified:
Animportantlimitationofthetwocasestudieswehave
• Delaysinbusarrivals(N=41),e.g.“Bus39at8:20a.m. presentedsofaristhatwhiletheyrelyonubiquitousandmo-
was10minlate”, biletechnology,theydonottakefulladvantageoftheubiqui-
• Driver behaviour (N = 84) such as being impolite, driv- touscapabilitiesofthesetechnologies.Forinstance,nocon-
ing in a dangerous manner, or showing no respect for textualinformationwaspresentedtoparticipants,eitherin
J.Goncalvesetal./ComputerNetworks90(2015)34–48 41
Fig.3. Left:theZoom/Pictureconditionviewingapictureofareportaboutaninaccessiblebuilding.ThePictureconditiondoesnothavethezoomwidget,the
Zoomconditiondoesnotshowphotographsuploadedbyotherusers,andtheControlconditionhasbothtypesofrestrictions.Right:addinganewreportwas
identicalacrossallconditions.
responsetotheiractionsoractionsofothers.Forthisreason, • seetheirownandothers’reports:dependingonthecon-
inthiscasestudyweseektoassesswhetherthepotentialof dition, the participant was shown or not shown a pho-
ubiquitoustechnologiesinprovidingcontextualinformation tographoftheinaccessiblelocation,andcouldorcould
canactasamotivatorforusers.Thiscasestudyisextensively notzoomintothemaptogetgranularinformationabout
describedin[22],andherewesummarisethemotivational theexactlocation.Participantscouldalwaysseethead-
aspectsofthiswork. dressandcommentsaddedtoanygiveninaccessiblespot
To investigate the effect of contextual cues on ubiqui- (Fig.3a);and
touscrowdsourcing,webuiltanonlinemap-basedplatform • addareportusingaformwhichwasidenticalforallpar-
whereparticipantscouldreportinaccessiblespotsthroughout ticipants.Toaddareportparticipantscouldzoomtoguar-
thecity. anteeaccuratepinpointingofthemarker,ratethesever-
Rather than create a map-based application for venues ityoftheinaccessiblespot(low–greenmarker,medium
andevents–whichalreadyexistcommercially–weoptedto – yellow marker, high – red marker), leave a message,
buildanonline“crowd-mapping”service,whichwouldulti- anduploadapictureofthelocation,whichallthesewere
matelyresultinanaccessibilitymapofthecity(Fig.3). mandatoryinallconditionsforconsistency(Fig.3b).
FollowingthefindingsfromCaseStudy2,wewantedto
3.3.1. MotivationalApproach
makesurethatparticipantswereabletovisualizetheircon-
Thiscasestudyaimedtoassesshowdisplayingdifferent
tributions and hopefully motivate them to further partici-
contextualinformationcanaffectthepersuasivepowerofa
pate. Those in the Control and Zoom conditions were able
message,andcontributetochangingtheattitudesthepartic-
toseetheirownpicturesbutneverthosesubmittedbyother
ipantsregardingcities’accessibility.
participants.Wedecidedtoallowthisastoavoidparticipants
We hypothesized that the presence of contextual infor-
tobegintoquestionthevalueoftheirphotographsandef-
mationwillleadto:
fortsinthesystem.Thiswasparticularlyimportant,asup-
• Increasedparticipation(morereportssubmitted)through loadingapicturewasoneoftherequirementstobeableto
recollectionandreflectionassuggestedbypreviousstud- submitareport.Finally,theControlconditionservedasan
ies[33]. exampleofsystemthatprovidesfeedbackandinformation
• Increasedawarenessofenvironmentalbarriersandinac- outofcontext.
cessiblespots[51].
3.3.2. Results
Wemanipulatedtwovariables: Intotal,wehad24participantsandreceived154reports,
23(14.9%)intheControlcondition,26(16.9%)intheZoom
• thepresenceoflocationcues(i.e.,abilitytozoom-intoan
condition,50(32.5%)inthePictureconditionand55(35.7%)
exactlocationonamap),and
in the Zoom&Picture condition. In terms of the severity of
• thepresenceofvisualcues(i.e.,showingin-mapphotos
inaccessiblespotsofthereportswereceivedtherewere39
whenbrowsingamap).
(25.3%)low,62(40.3%)medium,and53(34.4%)high.
This led to a 2 × 2 design with 4 conditions: Control, We found a significant effect of motivation on the total
Zoom,Picture,Zoom&Picture.Weinstructedparticipantsto numberofreportssubmittedbyparticipants(F(3,20)=.52,
takepicturesofinaccessiblelocationsaroundtowntoserve p=.67).ParticipantsinthePictureandZoom&Picturecondi-
as“proof”andthenlaterlogintoouronlineservicetoupload tionscontributedsignificantlymorereportswhencompared
theirreports.Eachparticipantwasallocatedtooneoffour tothoseintheControlcondition(p=.04andp=.02,respec-
conditionsthatmanipulatedtheuserinterfacedesignandin- tively).ParticipantsintheZoomconditiondidnotsubmitsig-
teractionmechanismsavailableinourwebapplication.After nificantlymorereportsthanthoseintheControlcondition.
logginginparticipantswerepresentedwithaGoogleMaps Tosubmitareport,participantshadtoratetheseverity
interfacewhichhadallreportsfromallconditions.Partici- ofthespot’sinaccessibility(low,medium,high).Wefounda
pantscould: significantrelationshipbetweenconditionandtheseverity
42 J.Goncalvesetal./ComputerNetworks90(2015)34–48
Fig.4. Settingofourdeployment.Left:conceptimageaftertherenovationgetscompleted.Middle:oneofthepublicdisplaysused.Right:sectionofthepedes-
trianstreetgettingrenovated.
levelparticipantsreported(χ2=21.35,df=6,p<.01).Low
severityinaccessiblespots(greenmarkers)weremorepopu-
laramongparticipantsintheZoomconditionandtheControl
condition.Incontrast,wefoundthatthetwoconditionswith
pictureshadagreaterinclinationtoreportmediumandhigh
severityinaccessiblespots(yellow/redmarkers).
3.3.3. Take-away
In this case study we demonstrate that contextual cues
can significantly enhance user participation in a “crowd-
map”platformtoreportinaccessiblespotsinacity.Ouranal-
ysisshowedthatourmanipulationofdifferentlevelsofcon-
textualinformationhadtwosignificanteffects.First,visual
cues led to enhanced user participation through increased
numberofreports.Second,visualcuesleadtoabiggersense
ofurgency,whichmotivateduserstofindmoresevereacces-
sibilityissuesthatrequiremoreimmediateattention.
3.4. CaseStudy4:cansituatednessmotivateubiquitous
crowdsourcing?
A significant affordance of ubiquitous crowdsourcing is
theabilitytorecruitandattractpeoplein-situ.Forinstance,it Fig.5. Feedbackinterfacewhichincludedavirtualkeyboardandastream
ispossibletocollectopinionsonanissuedirectlyco-located ofrecentsubmittedcomments.
withtheusedtechnology.Inthiscasestudyweinvestigate
thisaffordanceinthescopeofamajorrenovationofacity
providingcontextualinformationworkedwellinCaseStudy
centre,whichincludedbuildingnewpavementandunder-
2,wehypothesizedthatbyallowingcitizenstoprovideinput
ground heating systems for two of the busiest pedestrian
onanissuethattheycanactuallyseeonthespotwouldwork
streets (Fig. 4). This renovation heavily affected pedestrian
asanimportantaddedmotivatorforcontribution.
flowsandeverydaybusinessinallthesurroundingareas,was
Thetestedsystemwasanapplicationforthepublicdis-
aheatedtopicinthecity,andwasreportedindozensofsto-
playsthatallowedcitizenstoratetheprogressofthereno-
riesinlocalnewspaperswhereitgarneredheavyattentionin
vation,andtoprovideopen-endedfeedback. Theinputwas
thediscussionsectionsbothforandagainsttheproject.Dur-
givendirectlyonthepublicdisplaysusinganon-screenkey-
ingthisrenovationweexploredtheuseofpublicinteractive
board.Wealsoincludedastreamofrecentsubmittedcom-
displays for collecting feedback and ideas from citizens on
ments following findings from Case Study 2 regarding the
therenovationproject.Specifically,welookedatwhattype
motivational importance of presenting users with the re-
of feedback was collected by the displays that were physi-
sults of their contributions (Fig. 5). This practice has been
callyclosetotherenovation(i.e.,“in-situ”)asopposedtothe
suggested for enhancing communication between commu-
displaysthatwerefartheraway.
nitymembersonpublicdisplays[62].Wehypothesizedthis
would motivate feedback submission because letting users
3.4.1. Motivationalapproach
(cid:3)(cid:3) observe others’ messages enhances sense of community, a
Inthiscasestudy,weused57 full-HDtouchscreendis-
strongmotivatorforparticipationinurbansettings[9].
plays fitted in weather-proof casings. Many of the displays
had been located in the vicinity of the renovation area al-
readyforseveralyearsandassuchhavegonebeyondnovelty 3.4.2. Results
to be an accepted part of the city infrastructure itself [47]. During the deployment we collected a total of 246
Weused12displays:5locatedontherenovatedstreets(e.g., text-based feedback. The five public displays located next
Fig.4),and7fartherawaybutstillinpopularlocations.As to the reconstruction were significantly more popular
J.Goncalvesetal./ComputerNetworks90(2015)34–48 43
Table1
Taskuptakeandnoise(%)forconditionswithmotivationpresentandnotpresentforallcasestudies.
Nomotivationpresent Motivationpresent Difference(%)
Uptake(perday) Noise(%) Uptake(perday) Noise(%) Uptake(perday)(%) Noise(%)
CaseStudy1 12 40 12 22 – −18
CaseStudy2 1.64 18 3.12 3 90 −15
CaseStudy3 9 6 13.86 1 54 −5
CaseStudy4 0.36 56 1.43 84 297 28
(F(1,11)=13.07,p<.01)thantheseventhatwerenot(N=196 largernumberofusersthatmisappropriatedtheprototype.
vs.N=50).Thisdifferenceismoreaccentuatedifwelookat Wereflectmoreontheseresultsinourdiscussion.
theaverageamountofcommentscollectedperdisplay(39.2
vs.7.1). 4. Discussion
Some of the comments collected in the public displays
nexttothereconstructionhighlighttheimportanceofhaving Thecasestudieswepresentedhaveallbeenvalidatedand
theabilitytoparticipatein-situafterexperiencingorwhile testedinfieldtrials.Suchamethodologyisparticularlyim-
observing the reconstruction such as: “More employees are portant when investigating motivational aspects of ubiqui-
needed,thisneedstobedonefaster.”,“Lookinggood,alsothe toustechnologies.Theurbanspaceitselfisarichyetchal-
newstagelooksnice!”and“It’sgreattoseetheCitydevelop- lengingenvironmenttodeployubiquitousinfrastructureand
ing!”. applicationsin[43].Severalconsiderations,includingthein-
Finally, the comment stream allowed people to pick up tertwinedsocialpracticesofthelocation,robustnessofthe
previouscommentsanddiscussthem,oftenbyagreeingand technology,abuse,vandalism,balancebetweenthedifferent
supplementingthem.Anexamplesequenceconsistsofthe stakeholders,andevenweatherconditionsmaycausecon-
followingmessages:“Wastingyearsbecauseofthissmallren- straints when deploying in the wild [13]. However, to gain
ovationiswaytoolong.”,“Also,pleaseaddmoreworkinghours, anunderstandingofhowtechnologyisusedandappropri-
itistakingtoolong.”,and“Yea,Ialsoreallyagreeonthat”.This atedbythegeneralpublic,deploymentinauthenticenviron-
suggeststhataddingthecommentstreamledtofurtheren- ments,orlivinglaboratories,ishighlybeneficial[55].
gagementwithothercitizens. InTable2wesummariseallcasestudiesandtheirfind-
ingsintermsofmotivationandperformanceinubiquitous
crowdsourcing.Eachcasestudyaddressesoneoftheques-
3.4.3. Take-away
tionsweposedatthestartofthispaperasfollows:
This case study evaluates the impact of situatedness in
feedback collection. Our results highlight the potential of
1. Canubiquitouscrowdsourcingworkwithoutpaymentand
context to motivate participation in an urban context with
explicit recruitment? Yes, we demonstrate that with ap-
the public displays located right next to the reconstruc-
propriatemotivationubiquitouscrowdsourcingwithout
tionelicitingmorecontribution.Furthermore,showingusers
explicitrecruitmentandpaymentcancompetewithon-
each other’s contribution can foster discussion and further
line crowdsourcing in terms of quantity and quality of
increaseparticipation.
contributionsaswellasattractingnewworkers.
2. Can psychological empowerment motivate ubiquitous
3.5. Motivationeffectivenessacrosscasestudies crowdsourcing? Yes, it elicits more positive types of
contributionandincreasedparticipation.
To compare the effectiveness of motivation across our 3. Can contextual cues motivate ubiquitous crowdsourcing?
case studies, we further calculated task uptake (average Yes,theycanincreaseparticipationandimprovetheat-
number of tasks performed per day) and noise (% of bad titudesofworkers.
qualitycontributions)whenmotivationwaspresentandnot 4. Cansituatednessmotivateubiquitouscrowdsourcing?Yes,
present(Table1).Wethencalculatedthedifferencebetween itelicitsincreasedparticipationandengagement.
thesemetricsacrosstheconditionsthateitherentailedmo-
tivationordidnot(Table1). 4.1. Increasingparticipationandimprovingcontribution
The results show that motivation increased task uptake quality
forCaseStudies2(90%),3(54%)and4(297%).However,for
CaseStudy4wemostlyattributethesubstantialincreaseto Whilealtruismshouldbeenoughofamotivatorsinceit
thefactthatdisplaysco-locatedneartherenovationaresitu- appealstopeople’sdesiretohelp[53],inourcasestudieswe
atedinacentralareaofthecity.AsforCaseStudy1,taskup- showthattypicallyitisnot.Acrossallourcasestudieswe
takeremainedthesameduetothedesignoftheexperiment showthatthecontrolcondition(i.e.theconditionwithno
whichentailedhavingthesameamountoftaskscompleted motivationalmanipulation)suffersseverelyintermsofpar-
oneachconditionattheendofthestudy.Furthermore,mo- ticipation:peoplearesimplylesswillingtoparticipate.
tivationreducedtheamountofnoiseinmostcasesstudies Forexample,inCaseStudy1wefoundthatappropriate
(1,2and3).Whiletheabsolutenumberofvalidcomments motivationalandfact-checkingmechanismsinthedesignare
was higher when motivation was present in Case Study 4 animportantprerequisiteforcollectingaccurateresponses
(32vs.22),theirphysicallocationendedupattractingamuch fromusers.Specifically,whenconsideringonlythosepeople