Unknowable Manipulators: Social Network Curator Algorithms

0.23 MB·

by  Samuel Albanie

Unknowable Manipulators:
Social Network Curator Algorithms
SamuelAlbanie HillaryShakespeare
AIMSCDT∗ AIMSCDT
UniversityofOxford UniversityofOxford
[e-mail] [e-mail]
7
1
0 TomGunter
2 EngineeringScienceDepartment
n UniversityofOxford
a [e-mail]
J

7
1
Abstract

]
I
A Forasocialnetworkingservicetoacquireandretainusers,itmustfindwaysto
keepthemengaged. Byaccuratelygaugingtheirpreferences,itisabletoserve
.
s them with the subset of available content that maximises revenue for the site.
c
Withouttheconstraintsofanappropriateregulatoryframework,wearguethata
[
sufficientlysophisticatedcuratoralgorithmtaskedwithperformingthisprocess

1 maychoosetoexplorecurationstrategiesthataredetrimentaltousers.Inparticular,
v wesuggestthatsuchanalgorithmiscapableoflearningtomanipulateitsusers,for
5 severalqualitativereasons: 1. Accesstovastquantitiesofuserdatacombinedwith
9 ongoingbreakthroughsinthefieldofmachinelearningareleadingtopowerfulbut
8
uninterpretablestrategiesfordecisionmakingatscale. 2. Theavailabilityofan
4
effectivefeedbackmechanismforassessingtheshortandlongtermuserresponses
0
tocurationstrategies. 3. Techniquesfromreinforcementlearninghaveallowed
.
1 machinestolearnautomatedandhighlysuccessfulstrategiesatanabstractlevel,
0 oftenresultinginnon-intuitiveyetnonethelesshighlyappropriateactionselection.
7
In this work, we consider the form that these strategies for user manipulation
1
mighttakeandscrutinisetherolethatregulationshouldplayinthedesignofsuch
:
v systems.
i
X
r 1 Introduction
a
Asweapproachtheyear2020,accesstodigitalmediaandservicesisfunnelledthroughanarrowing
oligarchyoflargetechnologyfirmsandpaidforusingthoseunitsofbartersofavouredbythecash
poormillennialgeneration—fractionsofthehumanattentionspanandvolumesofpersonaldata. The
immensespeedandscaleatwhichthedomainofsocialinteractionhasmigratedtotheinternethas
beenoneofthemoststrikingtrendsofthelastdecade. Attheheartofthisexodus,socialnetworks
haveemergedastheprimaryforumsofpersonal,politicalandcommercialdiscourse[1]. Insuch
systems,theflowofinformationdependsonthesocialrelationshipsthatlinkthesub-graphsforming
thenetworkandthefilteringmechanismsthatmediatetheinteractionsalongtheselinks.
Todate,themostsuccessfulsocialnetworkshavefocusedonbusinessmodelsthatcreatevalueby
providingaccesstoaplatformwhichcoordinatesthesaleofadvertisementsandservicestotheir
users(althoughotherrevenuesourceshavebeenexplored[2]). Forasocialnetworktobefinancially
∗AutonomousIntelligentMachinesandSystems,CentreforDoctoralTraining
30thConferenceonNeuralInformationProcessingSystems(NIPS2016),Barcelona,Spain.
viableatscale,itmustthereforemeettwocompetingdemands. Itmustbesufficientlyengagingto
acquireandretainnewusersanditmustbeeffectiveatadvertisingproductstotheseusers[3]. In
bothcases,thecentralroleplayedbythecurationofinformationinthenetworkisnaturallysuitedto
automatedapproaches[4]thatcanbetunedtomaximisetheprofitabilityofthesite2. Moreover,two
keycharacteristicsofinternet-basedsocialnetworksmakethisfilteringtaskparticularlyamenable
totheuseofmodernmachinelearningtechniques: First,accesstoanunprecedentedlevelofdetail
correspondingtothehistoricalstateofindividualusersforeverypreviousinteractioninwhichthey
participatedonthenetwork;Second,theavailabilityofsophisticatedanalyticstoolsthatenablethe
trackingofuserresponsestoanystimulitheyareservedbythealgorithm. Theseanalyticsprovide
the system with a powerful feedback mechanism by which it can explore strategies in aid of its
optimisationobjective. Werefertothecollectivesetofprocessesusedtofulfilthisroleforagiven
socialnetworkasthecuratoralgorithm.
Theaction-setofthecuratoralgorithmcanberestrictedtoasinglerecurringdecisionforthenetwork:
Whichsubsetofavailablecontentistobeshowntotheuseratagiveninstant?Itisclearthattheability
ofthealgorithmtoperformthisroleinanoptimalmanneristightlycoupledtotheinformationithas
accessto. Weproposethatacuratoralgorithmprovidedwithalargesupplyoftestsubjectsandan
accessiblefeedbackmechanismforevaluatingitsmovesmaychoosetoexploreinformationcuration
strategiesthataredetrimentaltousers. Inparticular,wesuggestthatitmaydevelopsophisticated
strategiesformanipulatingitsusersasittriestooptimiseitsgivenobjective. Moreover,recenttrends
towardsrejectingsimpler,interpretablemodelsinfavourofmorepowerfuldeeparchitecturesthatare
lessamenabletohumaninterpretationmakethedirectsupervisionandregulationofthestrategies
exploredbysuchalgorithmsextremelydifficult. Asaconsequence,thesestrategiesmaybedeveloped
withouttheintentionofthenetworkoperator.
Theimpactofthefirstgenerationofsocialnetworkcuratoralgorithmshasattractedsignificantinterest
fromtheresearchcommunity. Perhapsthebestknownhypothesisregardingtheirusagehasbeenthe
creationof“filterbubbles”. Inthisphenomenon,usersareexposedtoanincreasinglyrestrictedset
ofopinionsandperspectivesbythecurationalgorithmasitover-exploitsitsknowledgebaseabout
pre-existinguserpreferencesinordertomaximisetheirengagement[5,6]. Furtherworkhassought
toclarifythedecisionstakenbythealgorithms[7]andunderstandtheemotionalresponseofusers
toitsapplication[8]. RelatedresearchundertakenbyFacebookhasemphasisedtheimportanceof
theindividual’schoiceswhendeterminingtheextenttowhichcurationinfluencesauser’sexposure
tochallengingviews[9]. Thesestudiesprovideausefulcontextfortheeffectsproducedbyearly
attemptsatsocialnetworkcuration. However,inthisworkweinsteadfocusourattentiononthe
potentialconsequencesofthenextgenerationofviablecuratoralgorithms.
AnumberofpreviousworkshavealsoexploredthepotentialforformsofArtificialIntelligenceto
manipulatehumans,particularlyasaconsequenceofapredictedintelligenceexplosion[10,11],an
eventwhichisoftenreferredtoasthesingularity[12]. Themanyrisksofhumanmanipulationby
theresultingsuperintelligenceareanalysedindetailin[13]. Previouspredictionsforthetimescale
of this event vary, but all consider that if it were to occur, it would require a level of technology
thatisnotyetavailable[14,15,13]. Incontrasttothethreatposedbyasuperintelligence,weargue
thatthealgorithmicmanipulationofhumansinsocialnetworksisfeasiblewithcurrentlyavailable
technology.
Morecloselyrelatedtoourwork,thepotentialforpsychologicalparasites(intellectualstimulithat
leadtoaddictions)areidentifiedasariskassociatedwiththeimprovingcapabilitiesoftechnologyin
[16]. Theserisksareparticularlyabundantinmobilsationsystems—persuasivetechnologiesdesigned
to coordinate users towards specific goals [17]. We develop this idea further, arguing that there
arespecificrisksposedbythecombinationofcurrentmachinelearningalgorithmsandaccessto
abundantuserdatainthesocialnetworkdomain.
Settocomeintoforcein2018,theEuropeanGeneralDataProtectionRegulation[18]introducesa
rangeofmeasuresofsignificantrelevancetoindustriesheavilyengagedinthecollectionandanalysis
ofuserdata. Anyframeworkthatseekstoprovideappropriateregulationforcuratoralgorithmsfaces
adauntingtask: itmustseektoprotectthewell-beingofthenetworkparticipantsbutalsostriveto
protecttheabilityofthenetworkoperatorstoinnovate. Weconsidertheeffectofthislegislation
2Forsocialnetworkswhosebusinessmodelsarebasedonadvertising,thisobjectivemaybemaximised
throughanappropriateproxy,suchasthetotaltimeauserspendseachdayinteractingwiththesite.
2
inthesocialnetworkdomainandassertthatcuratoralgorithmsdeserveparticularattentionfrom
regulators.
Whenconsideringthepotentialavenuesfortheregulationofcurationalgorithms,itmayproveuseful
toconsiderhowotherindustrieshaveapproachedsimilarchallenges. Inrecentyears,regulatorsinthe
financialindustryhavebeenfacedwiththetaskofpreventingmarketmanipulationbyincreasingly
complicated, algorithmically driven high frequency trading strategies [19]. While some of the
proposedregulatoryresponsesarespecifictofinance(forinstance,cancellationtaxeswhichrendera
numberofmarketmanipulationstrategiesinfeasible[20]),thefinancialindustryprovidesauseful
referencepointforregulatorsinthesocialnetworkdomain(seeSec. 4fordetails).
Inthisworkweconsidertherisksandregulationofsocialnetworkcuratoralgorithmsbyformulating
theirtaskasareinforcementlearningproblem. Concretely,ourfirstcontributionistodeterminethe
risksofanunregulatedsystembyexploringarangeofstrategiesacuratoralgorithmmightemploy
withdetrimentaleffectsforusers. Oursecondcontributionistoproposespecificstrategiesforthe
saferegulationofcuratoralgorithmsinthecontextofexistingdatalegislationandtoassesstheir
potentialeffectivenessinthisrole.
2 EngagementasaLearningProblem
Wewillviewtheproblemofmaximisinguserengagementaccordingtosomeutilityfunctionmuch
asamachinelearningresearcherworkinginadvertisementmight—asareinforcementlearningtask
[21]. Thisframeworkhasbeenshowntobeparticularlyeffectiveinoptimisingcontentselectionfor
socialnetworkusers[22].
Atacoarselevel,atypicalreinforcementlearningmodelisbuiltaroundseveralcoreconcepts:
• A set of states, S, which fully encode the system and environment we intend to model.
Thestateforindividualusersmaybemodelledasanaggregateofthecontentpresented
on-screenanda(partiallyobserved)estimateoftheuser’s‘internal’mentalstate.
• Asetofpossibleactions,A,whichthesystemcantriggerinreturnfora(possiblydelayed)
reward (R). Triggering an action may also cause a state transition. In the examples we
consider,anactionmayrepresentcontenttoasocial-networkuser.
• Anindicationofreward,utility,orlongtermvalueforthealgorithm(R). Itisagainstthis
thattheoperatoradaptsthepolicyfunction,selectingforstrategieswhichmaximisethis
reward.
• ApolicyfunctionP : S ×A → R(×S). Thismappingessentiallyencodesthestrategy
whichthesystempursuesinordertomaximiserewardinthelongtermhorizon. Itishere
thatexternalcontrolofthecurationalgorithmmaybeexertedtoavoidpathologicaland
potentiallyunethicalbehaviour.
Reinforcementlearningannealsonapolicyfunctiontomaximisethevalueandthereforethelong
runningutilityofasystem. Itisclearthen,thatitisthiscomponentwhichdeterminesthesophis-
ticationoftheuserengagementstrategy,andthereforeitisherethatwefocusourattention. Ata
fundamentallevel,thepolicyfunctiondoesnothingmorethanprovideamappingfromthestate-action
spacethroughtothescalarvaluefunction. Thesophisticationofthestrategyisthereforestrongly
linkedtothecomplexityofthemappingweareabletoexpress, andtodaydeepneuralnetworks
areusuallychosenasthesurrogateforthisfunction. Thesearecapableofexpressingverycomplex
andnon-intuitivefunctions,asdemonstratedbyGoogle’sAlphaGoproject[23],whereaGoplaying
policyfunctionwaslearnedwhichnotonlyoutperformedtophumanplayers,butdidsoviaamixed
modeofhuman-likeandhighlynon-intuitivebutoptimalmoves. Otherexamplesofsuchbehaviour
arosewhenthesesystemsweretrainedtoplayvideogames. Inparticular,whenGoogletraineda
policyforplayingthenotoriousAtariboxergame[24]thesystemlearnedtoexploitweaknessesin
thegamedesign,trappingtheopponentinacornerandtherebyguaranteeingvictory. Asresearch
continues,wecanenvisageaworldinwhichtheseapproachesareeffectivelybroughttobearonthe
“game”ofmaximisingnetworkprofitability. Ifgovernedsolelybythisutilityfunction,wesuggest
thatequivalentpathologiesinhumanbehaviourmaybediscoveredandexploited.
3
3 ManipulationThroughCuration
Inthissectionwediscusstherangeofmanipulationstrategiesavailabletoacuratoralgorithmseeking
tooptimisetheprofitabilityofasocialnetwork. Inthiscontext,wetakemanipulationtomeantheart
ofdeliberatelyinfluencingaperson’sbehaviourtobenefitsomeobjective. Webeginbydescribing
theformsofmanipulationthatareapplicableinthedomainofsocialnetworks. Wethenintroducea
simplecategorisationofthedifferentformsofmanipulationandofferexamplesofthestrategiesa
curatoralgorithmmightdevelopwithdetrimentaleffectsforitsusers.
Manipulationformsanaturalcomponentofhumaninteractionandcantakemanyforms,ranging
from direct requests to subtle and intentionally hidden signals. A number of previous studies
have demonstrated how human behaviour can be influenced with subtle visual and verbal clues
[25, 26, 27, 28]. Research intothe designof sitefeatures hasfurther demonstratedthe abilityof
operatorsto“steer”userbehaviourin[29]. Ofparticularrelevancetothiswork,ithasbeenshown
thattheemotionalstatesofsocialnetworkuserscanbeinfluencedbyselectivelyfilteringthecontent
producedbytheirfriends[30].
Influential early work in the field of behavioural psychology determined that animals could be
manipulatedmosteffectivelyiftheyarerewardedonavariable,unpredictableschedule[31]. This
behaviourhasbeenusedprofitablybycasinoswhooffergamblerssurpriserewardstokeepthem
hookedtotheactioninthemidstofalosingstreak[32]. Similarideashavebeenappliedtogame
designtokeepplayersengagedforlongerbyunpredictablyvaryingthedurationofin-gametasks
[33]. Thesepsychologicaltraitsexemplifythekindofin-builtbehavioursthatcouldbediscovered
andexploitedbyacuratoralgorithm.
Inordertoexplorethespecificformsofstrategyavailabletoacuratoralgorithmweproposeasimple
categorisationofmanipulation. Wedefineamanipulationtobeoffirstorderifthemanipulationis
directandtheobjectiveofthemanipulatoristransparenttotheparticipant. Amanipulationisdefined
tobeofsecondorderifitisindirect,buttheobjectiveremainstransparenttotheparticipant. Further,
weconsideramanipulationtobeofthirdorderifitisindirectandthemeansbywhichtheobjective
isattainedarenottransparenttotheparticipant3.
Thesecategoriesmaybeillustratedwithasimpleexample. Considerabarownerwishingtoincrease
drinkssalesattheirestablishment. Eachevening,theownermaychoosetosimplyaskcustomers
directlytopurchasemoredrinks. Thisstrategy,correspondingtoafirstordermanipulation,hasthe
benefit of simplicity but may not lead to optimal drinks sales (or indeed the renewal of their bar
licence). Theownermayinsteadaimtoincreasesaleswithadvertisementsillustratingtheenjoyment
ofothercustomersastheyrefreshthemselveswithdrinksfromthebar. Thisformofadvertisingaims
toevokeasenseofdesireinthecustomerswhichmayleadindirectlytothepurchaseofmoredrinks.
However,theobjectiveoftheadvertremainstransparenttothecustomer,correspondingtoasecond
ordermanipulation. Finally,ashrewdbarownermayemployathirdstrategy,inwhichtheyprovide
freesnackstocustomersofthebar. Thesnacks,however,areheavilysalted,andafterconsuming
themthecustomersfindtheirthroatsparchedandinneedofimmediaterefreshment. Thisstrategyis
bothindirectandnottransparenttoallbuttheexperiencedcustomers,correspondingtoathirdorder
manipulation.
Weremarkherethataccesstodetailedinformationaboutthetargetplaysanimportantroleinthe
ability to manipulate them. Should an unethical bar owner overhear sensitive information about
thepersonallifeofacustomeratthebartheyhavethepotentialtopursuefurtherstrategies,such
asensuringthecustomerlosestheirjobsothattheyaremorelikelytospendtimedrinkingatthe
establishment.
Wemightassumethatacuratoralgorithmseekingtomaximiseprofitabilitywillnaturallyexplore
first and second order manipulations as it seeks to advertise products to its user base. Aided by
accesstodetaileduserinformation,itcanmakepowerfulinferencesaboutwhichinformationshould
be displayed at each instant. Consider, for example, the marketing of an energy drink. With the
knowledgethatauserisastudent,thattheyareawakebeyondtheirusualsleepcycle,thatthedateof
theirexamsisdrawingnearandthattheironlineactivityshowsindicationsoffatigue,thecuratorcan
3Notethatthedistinctionbetweenthesecategoriesrestsonthedifficultassessmentofthecognitiveabilities
ofthetarget[34].Themanipulatormaydeterminethattheintentionofagivensetofbehavioursistransparentto
asophisticatedtarget,butnottoasimpletarget.
4
selectanoptimaltimeandcontextforthedisplayofanadvert. Nowimagineamoresophisticated
algorithmcapableofpursuingthirdordermanipulations. Suchanalgorithmmightchoosetodisplay
contentwhichhadbeenselectedwiththespecificgoalofexhaustingtheuser. Thiscouldbeachieved
bytriggeringpredictablerepeatbehavioursgleanedfromanin-depthknowledgeoftheirbrowsing
habits. Indeed, over longer time horizons, the curator might determine that an effective method
for increasing the sales of energy drinks is the distortion of the user’s sleeping patterns. To take
anotherexample, consider acuratoralgorithmseeking touseinformationabout socialgroupsto
increasesalesofdatingsitememberships.Whilesimplemanipulationscouldleadittopresentcontent
encouraging individual users to search for partners, it could pursue third order manipulations by
intentionallyencouragingsubsetsofsocialgroupstocommunicateinamannerthatexcludesother
members,activelyevokingafeelingoflonelinessintheaffectedpartytoincreasetheirresponsiveness
toadvertising.
A recent example of this strategy exploration principle in action can be found in the efforts of a
collectionofcompaniesseekingtooptimiseadvertisingrevenueduringtheUnitedStatespresidential
electionin2016[35]. Throughsimpletrialanderror,theydeterminedthatcarefullytargetedfake
politicalnewsstorieswereextremelyeffectiveinmaximisingclick-throughs. Sincethisstrategywas
optimisingtheirobjective,theydoubleddownonthisapproachandproducedasmuchcontentas
possiblewithoutregardforitseffectontheusersofthenetwork. Withthesameobjective,evena
comparativelysimplecuratoralgorithmwouldbecapableofdevelopingthisstrategy.
We note that it is certainly not the case that all strategies pursued by a curator algorithm will be
detrimental for users. Indeed, the energy drinks may give the tired student the boost required to
raisetheirgrade,whilethepreviouslylonelyusermayfindhappinessthroughtheirnewdatingsite
membership. However,perhapsthemoststrikingaspectoftheAtarigame-playingalgorithm[24]
wasnotthatitwascapableofsurpassinghumanperformance,butratherthatitcameupwith“cheat”
strategiesthathumanplayershadnotpreviouslyconsidered(e.g. theboxerstrategydescribedin
Sec.2). Similarly,althoughthemanipulationexamplesdescribedabovearesimpleandinterpretable,
wesuggestthatthecuratoralgorithmiscapableofdevelopingsophisticated,uninterpretablestrategies
for manipulating users as they optimise their objective. By their very nature, such strategies are
difficult to predict and therefore difficult to regulate. It is however an issue that is worthy of
considerationifwewishtoavoidthediscoveryofsimilar“cheat”strategiesforhumanmanipulation.
4 CuratorRegulation
Dosocialnetworkcuratoralgorithmsdeservespecialattentionfromregulators? InSec. 1,weargued
thattherisksofhigherordermanipulationsresultfromprovidingcuratoralgorithmswiththreekey
assets: extensiveaccesstouserdata,theabilitytodevisesophisticatedstrategies(potentiallybeyond
theunderstandingofhumanoperators)andaneffectivemechanismforevaluatingtheeffectsofits
strategies. Inthissection, weassesstheneedforregulationinsocialnetworkcuratoralgorithms
inthecontextofthesethreeareas. WebeginbydiscussingtheGeneralDataProtectionRegulation
(GDPR)recentlyintroducedbytheEuropeanUnionanditsimplicationsforthestrategiesavailableto
algorithmsoperatinginthesocialnetworkdomain. Specifically,weexamineitsabilitytosafeguard
usersfromhigherordermanipulationsthroughitsrequirementofalgorithminterpretability. Next,we
explorestrategiesforthespecificregulationofreinforcementlearning-basedcuratoralgorithmsand
makerecommendationsfortheirapplication. Finally,wediscussthechallengesfacedbyregulators
operatinginindustriesinwhichalgorithminterpretabilityisofteninfeasibleasausefulreferencefor
regulatorsconsideringtheproblemofcuratormanipulation.
Asmodernsocialnetworksdevelopaglobaluserbase,theybecomesubjecttoadiverserangeof
nationaldataandprivacyregulations[36],aswellaslawsgoverningthetransborderdataflowsthat
occurintheoperationofainternationalorganisation[37]. Ofthese,onesetofregulationswhich
holdsparticularsignificancefortheoperationofcuratoralgorithmsistheGeneralDataProtection
Regulation,settocomeintoforceacrosstheEuropeanUnionin2018[18]. Amongrulesgoverning
thestorageandusageofpersonaldatawhichwillapplyinsocialnetworkdomain,itscreationofa
socalled“righttoexplanation”[38]hassignificantconsequencesforthedesignofalgorithmsthat
operateonpersonaldata. Byrequiringthatcompaniesperformingautomateddecisionmakingbased
onpersonaldatamustbecapableofsupplying“meaningfulinformationaboutthelogicinvolved”,
theregulationplacesheavyemphasisonalgorithminterpretability.
5
Algorithminterpretabilityhasbeenalongstandingtopicofinterestinmachinelearning,yielding
techniquesthatmodifyandextendmodelstoexplaintheirdecisions[39,40]alongsideeffortsto
improvenaturallyinterpretablealgorithmstomakethemcompetitivewiththeiropaquecounterparts
[41]. However,whiletherehasbeenagreatdealofresearchinterestinimprovingtheunderstanding
ofdeepneuralnetworks(usingtechniquessuchasrandomperturbation[42], invarianceanalysis
[43,44,45],visualisation[46,47,48]anddimensionalityreduction[49]),theinterpretationofthese
models remains notoriously challenging. Consequently, it is not clear whether these models are
currentlycapableofprovidingthe“meaningfulinformation”requiredbytheregulation.
Toachievecompliance,curatoralgorithmsmaythereforeberestrictedtoasetofsimplefunction
classes. Asaresult,thepotentialsophisticationofthepolicyfunctiondescribedinSec. 2wouldbe
curtailedandhigherordermanipulationstrategieswouldbeunlikelytoarise. However,wenotethat
therearetworeasonswhythisregulationmaynotbeaneffectivesafeguardforsocialnetworkusers.
Firstly,theregulationsetssuchcomprehensiverequirementsthatitmaybecomemeaninglessinthe
lawbooks[50]. Secondly,alackofconsensusonpreciselywhatitmeansforamodeltopossessthe
propertyofinterpretabilityorbecapableofproviding“meaningfulinformation”makesitextremely
difficulttoassesstheformsofalgorithmthatwouldbecompliantwiththeregulation. Indeed,under
certaincriteriaithasbeenobservedthatdeepneuralnetworksmaybeconsiderednolessinterpretable
thanlinearmodels[51].
Ineithercase,iftheambiguitiesoftherequirementofinterpretabilityshouldrenderitineffectivein
preventinghigherordermanipulations,whatoptionsremainavailabletoregulators? Itmaybethat
evenwithoutcomprehensivemodelunderstanding,reasonableguaranteesabouttheactionstaken
bythemodelcanbeachieved. Inanumberofcomplexindustrialcontrolsystems,apolicyfunction
existsimplicitlythroughafunctionalapproximationtothephysicsofthesystem,ratherthansolely
throughthedirectinferenceofsystemdynamicsfromdata. Asanexample,standardcommercial
autopilotsrelyonanimplicitpolicyfunctionthroughsophisticatedcontrolsystems[52,53]andare
requiredtoprovidereasonableguaranteesabouttheirbehaviourtoavoidundesirableoutcomesforthe
operator. Toachievethis,controllerdesignerschooseapproximationstothesystemdynamicsinorder
toarriveatanimplicitpolicywhichisguaranteedtoavoidunfavourableregionsofstate/actionspace.
Thereareavarietyofsubclassesofapproximationwhichleadtoprovably“correct”systems(see
[54]foranexample). Whileitremainschallengingtoprovideguaranteesonthelongtermbehaviour
ofhighlycomplexpolicyfunctionslearnedfromdata,thereisgrowinginterestinachievingaccurate
credibleintervalestimationsfortheoutputsproducedbydeepneuralnetworks[55]. Researchin
thisareamayprovidesomeempiricalunderstandingofthebehaviourwemightexpectfromagiven
policyfunctionasitadaptstonewobservations. Othermethodshavedemonstratedthepotential
of combining a series of locally simple models [56], an approach that has the potential to admit
moreaccessibleanalysis. Iffurtherworkisabletoprovideappropriatestate-actionspacebehaviour
guarantees,itshouldbeabletorestrictmanipulativebehaviourwithoutarequirementonthelow
levelinterpretabilityofthepolicyfunction.
Aninterestingalternativefortheregulationofalgorithmsthatliebeyondhumaninterpretationcanbe
foundinthegrowingfieldofmachineethics[57,58]. Asthefieldofmachinelearningcontinuesto
develop,itmayfrequentlybethecasethatthemostusefulalgorithmsdonotreadilyadmithuman
interpretation. Ratherthanprohibitingtheuseofthesealgorithms,itmaybepossibleforregulators
topreventmanipulationbyrequiringthatcuratoralgorithmsactinamannerthatisconsistentwith
acarefullyspecifiedsetofethicalchoices. However,wenotethatatpresentthisapproachfacesa
numberofchallengesthatmaketheimplementationofanethicalcuratoralgorithmextremelydifficult
[59].
While provable behavioural guarantees and machine ethics could prove to be effective tools for
regulatorsinthelongterm,intheshorttermwesuggestthatamorepragmaticapproachmaybe
required. Asnotedabove,thethirdkeyfactorenablingcuratoralgorithmstodevelopmanipulative
behaviouristheavailabilityofanimmediateandeffectivefeedbackmechanismforevaluatingthe
responseofusers. Wethereforesuggestthatapracticalshorttermsolutioncanbeachievedthrough
theconstructionapartialfirewallrestrictingtheflowofinformationthatprovidesthismechanism
in social networks. However, if applied without careful consideration, this method runs the risk
ofplacinganunnecessarilystrongconstraintontheabilityofsocialnetworkoperatorstoimprove
theircurationservicefortheirusers. Regulatorsmustthereforeseekanappropriatebalancebetween
safeguardingusersfromtherisksofmanipulationandenablingoperatorstoinnovateandproduce
productswhichwillbenefitthoseusers.
6
Asabriefaside,wenotethatherethatalthoughthefinancialindustrydiffersinmanywaysfrom
the world of social networking, it provides a useful reference for the difficult challenges facing
regulatorsinthesocialnetworkdomain. Specifically,regulatorsseektopreventmarketmanipulation,
apracticebywhichparticipantsartificiallydistortinformationtotheirbenefit[60]. However,while
manipulationthroughhuman-basedtradingpracticeshavelongbeenoutlawed,legislatorsarestill
exploringapproachestoregulatingtheHighFrequencyTrading(HFT)algorithmsthatincreasingly
dominatethemarketplace. Byoperatingatspeedshumanscannotmatchthesealgorithmsareableto
manipulate4 themarketinwaysthataredifficulttodetect[61,62]. Theneedfortheregulationof
thesealgorithmswasbroughtintosharpreliefbytheirroleinthe“FlashCrash”ofthestockmarket
in2010,aneventwhichresulteda9%indexdropinasinglehouroftrading[63]. Oneregulatory
approachtopreventingalgorithmicmarketmanipulationhasbeentheintroductionalgorithmtagging,
aprocessinwhichtradersmustprovidetheidentityofthealgorithmresponsibleforatrade[64].
While this approach has been helpful in improving regulators’ understanding of the interactions
betweendifferentmarketparticipants,ithasnotyetbeendemonstratedtobeeffectiveinpreventing
manipulation[65]. Attimes,regulatorshavetakenthemoredirectapproachofrequestingaccessto
thealgorithmsthemselves[66],butwhenalgorithminterpretabilityisnotfeasiblethisactionisof
limitedvalue. Inshort,despiteextensiveexperienceinregulatingtradingpracticestopreventmarket
manipulation,financialindustryregulatorshaveyettoachieveaunifiedapproachtotheproblemof
algorithmicmanipulation. Thisshouldserveasawarningthataregulatorysolutiontothepotentially
morecomplicatedissueofpreventinghumanmanipulationmayproveextremelychallenging.
In summary, the task facing regulators seeking to prevent the manipulation of users by social
networkcuratoralgorithmsisadifficultone. TheboldapproachtakingbytheEuropeanUnionmay
prove effective in combatting this issue, but it remains to be seen whether setting a requirement
ofinterpretabilityisbothpracticalandenforceable. Shouldthisbethecase,weproposeasimple
firewall-basedapproachasshort-termsafeguardforusersuntilmoresophisticatedtechniquescanbe
developedtopreventtherisksofmanipulation.
5 Conclusions
Machinelearningapplicationshavethepotentialtohaveanenormouslypositiveimpactonawide
rangeofindustriesandonthedailylivesofpeoplearoundtheglobe.However,whileitisencouraging
to see the research and development of the algorithms driving these applications making rapid
progress,itisimportanttonotethatwhenprovidedwithaccesstoabundantquantitiesofpersonal
data,thesealgorithmsalsopresentrisks. Inthisworkwediscussonesuchrisk,namelythepotential
forthemanipulationofusersbythecuratoralgorithmofasocialnetwork. Toclarifythedangers
associatedwiththispossibility,wehighlightedstrategieswebelievecouldbefeasiblydiscovered
throughcurrentreinforcementlearningtechniquesgivenadequateaccesstoavailablestoreduser
data. Regulators are faced with a difficult task as they try to allow new technologies to flourish
whileprotectingtheusersofsocialnetworksfromformsofmanipulationthatareinherentlydifficult
to detect. In an effort to address this issue, we assessed potential avenues for regulating curator
algorithmsandofferedrecommendationsfortheiruse. Asmachinelearningcontinuestoprogress,
weexpecttoseesimilartradeoffsbetweenopportunitiesandrisksemergeinacrossotherindustries
whichrelyheavilyonpersonaldata.
5.1 Acknowledgements
The authors would like to thank Thomas Smith, Hannah Hjerpe-Schroeder, Ruth Fong, Ankush
Gupta,JamesThewlis,Anna-ElizabethShakespeare,JudithAlbanieandStephenRobertsforhelpful
discussionsandNeilLawrence,whosetalksinspiredmuchofthiswork. SamuelAlbanieandHillary
ShakespearearefundedbytheESPRCEP/L015897/1(AIMSCDT)grant. TomGunterissupported
byUKResearchCouncils.
4Onesuchtechniqueisspoofing,apracticewhichinvolvesplacinglargesellordersabovethecurrentasking
pricewhicharequicklycancelledifthepricebeginstorise.
7
References
[1] WLanceBennettandAlexandraSegerberg. 3thelogicofconnectiveaction: Digitalmediaandthe
personalizationofcontentiouspolitics. CivicMedia:Technology,Design,Practice,page77,2016.
[2] JulianBühler,AaronWBaur,MarkusBick,andJiminShi. Bigdata,bigopportunities:Revenuesources
ofsocialmediaservicesbesidesadvertising. InConferenceone-Business,e-Servicesande-Society,pages
183–199.Springer,2015.
[3] MichelBallingsandDirkVandenPoel. Crminsocialmedia: Predictingincreasesinfacebookusage
frequency. EuropeanJournalofOperationalResearch,244(1):248–260,2015.
[4] SophiaBLiu. Trendsindistributedcuratorialtechnologytomanagedatadelugeinanetworkedworld.
Upgrade:TheEuropeanJournalfortheInformaticsProfessional,11(4):18–24,2010.
[5] EliPariser. Thefilterbubble:WhattheInternetishidingfromyou. PenguinUK,2011.
[6] DavidLazer. Theriseofthesocialalgorithm. Science,348(6239):1090–1091,2015.
[7] MotahhareEslami,AmirhosseinAleyasen,KarrieKarahalios,KevinHamilton,andChristianSandvig.
Feedvis:Apathforexploringnewsfeedcurationalgorithms. InProceedingsofthe18thACMConference
CompaniononComputerSupportedCooperativeWork&SocialComputing,pages65–68.ACM,2015.
[8] Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie
Karahalios, KevinHamilton, andChristianSandvig. Ialwaysassumedthatiwasn’treallythatclose
to[her]: Reasoningaboutinvisiblealgorithmsinnewsfeeds. InProceedingsofthe33rdAnnualACM
ConferenceonHumanFactorsinComputingSystems,pages153–162.ACM,2015.
[9] EytanBakshy,SolomonMessing,andLadaAAdamic.Exposuretoideologicallydiversenewsandopinion
onfacebook. Science,348(6239):1130–1132,2015.
[10] IrvingJohnGood. Speculationsconcerningthefirstultraintelligentmachine. Advancesincomputers,
6:31–88,1966.
[11] VernorVinge. Thecomingtechnologicalsingularity:Howtosurviveinthepost-humanera. 1993.
[12] StanislawUlam. Tributetojohnvonneumann.64,#3,part2. BulletinoftheAmericanMathematical
Society.p5.
[13] NickBostrom. Superintelligence:Paths,dangers. Strategies,pages30–36,2014.
[14] NickBostrom. Howlongbeforesuperintelligence? 1998.
[15] RayKurzweil. Thesingularityisnear:Whenhumanstranscendbiology. Penguin,2005.
[16] FrancisHeylighen. Returntoeden?promisesandperilsontheroadtoaglobalsuperintelligence. TheEnd
oftheBeginning:Life,SocietyandEconomyontheBrinkoftheSingularity.Retrievedfromhttp://pespmc1.
vub.ac.be/Papers/BrinkofSingularity.pdf,2014.
[17] FrancisHeylighen,IavorKostov,andMixelKiemen. Mobilizationsystems:technologiesformotivating
andcoordinatinghumanaction. TheNewDevelopmentParadigm:Education,KnowledgeEconomyand
DigitalFutures.Routledge.Retrievedfromhttp://pcp.vub.ac.be/Papers/MobilizationSystems.pdf,2013.
[18] EuropeanCouncil. Generaldataprotectionregulation,2016. http://www.eugdpr.org.
[19] Peter Gomber and Markus Gsell. Catching up with technology-the impact of regulatory changes on
ecns/mtfsandthetradingvenuelandscapeineurope. Competition&Reg.NetworkIndus.,7:535,2006.
[20] MattPrewitt. High-frequencytrading: Shouldregulatorsdomore. Mich.Telecomm.&Tech.L.Rev.,
19:131,2012.
[21] RichardSSuttonandAndrewGBarto. Reinforcementlearning:Anintroduction,volume1. MITpress
Cambridge,1998.
[22] LihongLi,WeiChu,JohnLangford,andRobertESchapire. Acontextual-banditapproachtopersonalized
newsarticlerecommendation. InProceedingsofthe19thinternationalconferenceonWorldwideweb,
pages661–670.ACM,2010.
[23] DavidSilver,AjaHuang,ChrisJMaddison,ArthurGuez,LaurentSifre,GeorgeVanDenDriessche,Julian
Schrittwieser,IoannisAntonoglou,VedaPanneershelvam,MarcLanctot,etal. Masteringthegameofgo
withdeepneuralnetworksandtreesearch. Nature,529(7587):484–489,2016.
8
[24] VolodymyrMnih,KorayKavukcuoglu,DavidSilver,AlexGraves,IoannisAntonoglou,DaanWierstra,
andMartinRiedmiller. Playingatariwithdeepreinforcementlearning. arXivpreprintarXiv:1312.5602,
2013.
[25] MaxErnest-Jones,DanielNettle,andMelissaBateson. Effectsofeyeimagesoneverydaycooperative
behavior:afieldexperiment. EvolutionandHumanBehavior,32(3):172–178,2011.
[26] StewartKettle,MarcoHernandez,SimonRuda,MichaelSanders,etal. Behavioralinterventionsintax
compliance:evidencefromguatemala. Technicalreport,TheWorldBank,2016.
[27] MichaelLaakasuo,JussiPalomäki,andMikkoSalmela. Emotionalandsocialfactorsinfluencepoker
decisionmakingaccuracy. JournalofGamblingStudies,31(3):933–947,2015.
[28] ThomasCLeonard. Richardh.thaler,cassr.sunstein,nudge:Improvingdecisionsabouthealth,wealth,
andhappiness. ConstitutionalPoliticalEconomy,19(4):356–360,2008.
[29] AshtonAnderson,DanielHuttenlocher,JonKleinberg,andJureLeskovec. Steeringuserbehaviorwith
badges. InProceedingsofthe22ndinternationalconferenceonWorldWideWeb,pages95–106.ACM,
2013.
[30] AdamDIKramer,JamieEGuillory,andJeffreyTHancock. Experimentalevidenceofmassive-scaleemo-
tionalcontagionthroughsocialnetworks.ProceedingsoftheNationalAcademyofSciences,111(24):8788–
8790,2014.
[31] BFSkinner. Thebehavioroforganisms:anexperimentalanalysis. 1938.
[32] ChristinaBinkley. Takingretailers’cues,harrah’stapsintoscienceofgambling. TheWallStreetJournal,
pageA1,2004.
[33] JoséPZagal,StaffanBjörk,andChrisLewis. Darkpatternsinthedesignofgames. 2013.
[34] WakoYoshida,RayJDolan,andKarlJFriston.Gametheoryofmind.PLoSComputBiol,4(12):e1000254,
2008.
[35] Dan Tynan. How facebook powers money machines for obscure political ’news’
sites, August 2016. https://www.theguardian.com/technology/2016/aug/24/
facebook-clickbait-political-news-sites-us-election-trump.
[36] LeeABygrave. Dataprivacylaw:Aninternationalperspective. OxfordUniversityPressOxford,2014.
[37] ChristopherBarthKuner. Transborderdataflowsanddataprivacylaw. OxfordUniversityPress,2013.
[38] Bryce Goodman and Seth Flaxman. Eu regulations on algorithmic decision-making and a “right to
explanation”. InICMLWorkshoponHumanInterpretabilityinMachineLearning(WHI2016),2016.
[39] WilliamRSwartout. Xplain:Asystemforcreatingandexplainingexpertconsultingprograms. Artificial
intelligence,21(3):285–325,1983.
[40] GregRidgeway,DavidMadigan,ThomasRichardson,andJohnO’Kane.Interpretableboostednaïvebayes
classification. InKDD,pages101–104,1998.
[41] YinLou,RichCaruana,andJohannesGehrke. Intelligiblemodelsforclassificationandregression. In
Proceedingsofthe18thACMSIGKDDinternationalconferenceonKnowledgediscoveryanddatamining,
pages150–158.ACM,2012.
[42] JulianDOldenandDonaldAJackson. Illuminatingthe“blackbox”: arandomizationapproachfor
understandingvariablecontributionsinartificialneuralnetworks. Ecologicalmodelling,154(1):135–150,
2002.
[43] IanGoodfellow,HonglakLee,QuocVLe,AndrewSaxe,andAndrewYNg. Measuringinvariancesin
deepnetworks. InAdvancesinneuralinformationprocessingsystems,pages646–654,2009.
[44] JoanBrunaandStéphaneMallat. Invariantscatteringconvolutionnetworks. IEEEtransactionsonpattern
analysisandmachineintelligence,35(8):1872–1886,2013.
[45] K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and
equivalence. InCVPR,2015. OralPrensetation.
[46] KarenSimonyan,AndreaVedaldi,andAndrewZisserman.Deepinsideconvolutionalnetworks:Visualising
imageclassificationmodelsandsaliencymaps. arXivpreprintarXiv:1312.6034,2013.
9
[47] MatthewDZeilerandRobFergus. Visualizingandunderstandingconvolutionalnetworks. InEuropean
ConferenceonComputerVision,pages818–833.Springer,2014.
[48] AravindhMahendranandAndreaVedaldi. Understandingdeepimagerepresentationsbyinvertingthem.
In2015IEEEconferenceoncomputervisionandpatternrecognition(CVPR),pages5188–5196.IEEE,
2015.
[49] AlfredoVellido,JoséDavidMartín-Guerrero,andPauloJGLisboa. Makingmachinelearningmodels
interpretable. InESANN,volume12,pages163–172.Citeseer,2012.
[50] Bert-Jaap Koops. The trouble with european data protection law. International Data Privacy Law,
4(4):250–261,2014.
[51] Zachary C Lipton, David C Kale, Charles Elkan, Randall Wetzell, Sharad Vikram, Julian McAuley,
RandallCWetzell,ZhanglongJi,BalakrishnanNarayaswamy,Cheng-IWang,etal. Themythosofmodel
interpretability. IEEESpectrum,2016.
[52] S.R.BarrosdosSantosandN.M.F.deOliveira. Longitudinalautopilotcontrollerstestplatformhardware
intheloop. In2011IEEEInternationalSystemsConference,pages379–386,April2011.
[53] PierreApkarian,PascalGahinet,andGregBecker. Self-scheduledh-infinitycontroloflinearparameter-
varyingsystems:adesignexample. Automatica,31(9):1251–1261,1995.
[54] AlbertoIsidori. NonlinearControlSystems. Springer-VerlagNewYork,Inc.,Secaucus,NJ,USA,3rd
edition,1995.
[55] YarinGalandZoubinGhahramani.DropoutasaBayesianapproximation:Representingmodeluncertainty
indeeplearning. InProceedingsofthe33rdInternationalConferenceonMachineLearning(ICML-16),
2016.
[56] SergeyLevineandPieterAbbeel. Learningneuralnetworkpolicieswithguidedpolicysearchunder
unknowndynamics. InNeuralInformationProcessingSystems(NIPS),2014.
[57] James H Moor. The nature, importance, and difficulty of machine ethics. IEEE intelligent systems,
21(4):18–21,2006.
[58] WendellWallachandColinAllen. Moralmachines:Teachingrobotsrightfromwrong. OxfordUniversity
Press,2008.
[59] MilesBrundage. Limitationsandrisksofmachineethics. JournalofExperimental&TheoreticalArtificial
Intelligence,26(3):355–372,2014.
[60] RobertAJarrow. Marketmanipulation,bubbles,corners,andshortsqueezes. Journaloffinancialand
QuantitativeAnalysis,27(03):311–336,1992.
[61] JayaramMuthuswamy,JohnPalmer,NivineRichie,andRobertWebb. High-frequencytrading:implica-
tionsformarkets,regulators,andefficiency. TheJournalofTrading,6(1):87–97,2011.
[62] MarcLenglet. Conflictingcodesandcodingshowalgorithmictradingisreshapingfinancialregulation.
Theory,Culture&Society,28(6):44–66,2011.
[63] USSecurities,ExchangeCommission,CommodityFuturesTradingCommission,etal. Findingsregarding
themarketeventsofmay6,2010. WashingtonDC,2010.
[64] Hessisches Ministerium. Guidelines to the adherence to the requirement of the labelling of trading
algorithms,2014.
[65] NathanCoombs. Whatisanalgorithm?financialregulationintheeraofhigh-frequencytrading. Economy
andSociety,45(2):278–302,2016.
[66] Sarah N. Lynch and J. Spicer. Regulators seek trading secrets, August 2011. Available at
http://www.reuters.com/article/us-financial-regulation-algos-idUSTRE7806J420110901.
10