Table Of Content

Unknowable Manipulators: Social Network Curator Algorithms SamuelAlbanie HillaryShakespeare AIMSCDT∗ AIMSCDT UniversityofOxford UniversityofOxford [email protected] [email protected] 7 1 0 TomGunter 2 EngineeringScienceDepartment n UniversityofOxford a [email protected] J 7 1 Abstract ] I A Forasocialnetworkingservicetoacquireandretainusers,itmustfindwaysto keepthemengaged. Byaccuratelygaugingtheirpreferences,itisabletoserve . s them with the subset of available content that maximises revenue for the site. c Withouttheconstraintsofanappropriateregulatoryframework,wearguethata [ sufficientlysophisticatedcuratoralgorithmtaskedwithperformingthisprocess 1 maychoosetoexplorecurationstrategiesthataredetrimentaltousers.Inparticular, v wesuggestthatsuchanalgorithmiscapableoflearningtomanipulateitsusers,for 5 severalqualitativereasons: 1. Accesstovastquantitiesofuserdatacombinedwith 9 ongoingbreakthroughsinthefieldofmachinelearningareleadingtopowerfulbut 8 uninterpretablestrategiesfordecisionmakingatscale. 2. Theavailabilityofan 4 effectivefeedbackmechanismforassessingtheshortandlongtermuserresponses 0 tocurationstrategies. 3. Techniquesfromreinforcementlearninghaveallowed . 1 machinestolearnautomatedandhighlysuccessfulstrategiesatanabstractlevel, 0 oftenresultinginnon-intuitiveyetnonethelesshighlyappropriateactionselection. 7 In this work, we consider the form that these strategies for user manipulation 1 mighttakeandscrutinisetherolethatregulationshouldplayinthedesignofsuch : v systems. i X r 1 Introduction a Asweapproachtheyear2020,accesstodigitalmediaandservicesisfunnelledthroughanarrowing oligarchyoflargetechnologyfirmsandpaidforusingthoseunitsofbartersofavouredbythecash poormillennialgeneration—fractionsofthehumanattentionspanandvolumesofpersonaldata. The immensespeedandscaleatwhichthedomainofsocialinteractionhasmigratedtotheinternethas beenoneofthemoststrikingtrendsofthelastdecade. Attheheartofthisexodus,socialnetworks haveemergedastheprimaryforumsofpersonal,politicalandcommercialdiscourse[1]. Insuch systems,theflowofinformationdependsonthesocialrelationshipsthatlinkthesub-graphsforming thenetworkandthefilteringmechanismsthatmediatetheinteractionsalongtheselinks. Todate,themostsuccessfulsocialnetworkshavefocusedonbusinessmodelsthatcreatevalueby providingaccesstoaplatformwhichcoordinatesthesaleofadvertisementsandservicestotheir users(althoughotherrevenuesourceshavebeenexplored[2]). Forasocialnetworktobefinancially ∗AutonomousIntelligentMachinesandSystems,CentreforDoctoralTraining 30thConferenceonNeuralInformationProcessingSystems(NIPS2016),Barcelona,Spain. viableatscale,itmustthereforemeettwocompetingdemands. Itmustbesufficientlyengagingto acquireandretainnewusersanditmustbeeffectiveatadvertisingproductstotheseusers[3]. In bothcases,thecentralroleplayedbythecurationofinformationinthenetworkisnaturallysuitedto automatedapproaches[4]thatcanbetunedtomaximisetheprofitabilityofthesite2. Moreover,two keycharacteristicsofinternet-basedsocialnetworksmakethisfilteringtaskparticularlyamenable totheuseofmodernmachinelearningtechniques: First,accesstoanunprecedentedlevelofdetail correspondingtothehistoricalstateofindividualusersforeverypreviousinteractioninwhichthey participatedonthenetwork;Second,theavailabilityofsophisticatedanalyticstoolsthatenablethe trackingofuserresponsestoanystimulitheyareservedbythealgorithm. Theseanalyticsprovide the system with a powerful feedback mechanism by which it can explore strategies in aid of its optimisationobjective. Werefertothecollectivesetofprocessesusedtofulfilthisroleforagiven socialnetworkasthecuratoralgorithm. Theaction-setofthecuratoralgorithmcanberestrictedtoasinglerecurringdecisionforthenetwork: Whichsubsetofavailablecontentistobeshowntotheuseratagiveninstant?Itisclearthattheability ofthealgorithmtoperformthisroleinanoptimalmanneristightlycoupledtotheinformationithas accessto. Weproposethatacuratoralgorithmprovidedwithalargesupplyoftestsubjectsandan accessiblefeedbackmechanismforevaluatingitsmovesmaychoosetoexploreinformationcuration strategiesthataredetrimentaltousers. Inparticular,wesuggestthatitmaydevelopsophisticated strategiesformanipulatingitsusersasittriestooptimiseitsgivenobjective. Moreover,recenttrends towardsrejectingsimpler,interpretablemodelsinfavourofmorepowerfuldeeparchitecturesthatare lessamenabletohumaninterpretationmakethedirectsupervisionandregulationofthestrategies exploredbysuchalgorithmsextremelydifficult. Asaconsequence,thesestrategiesmaybedeveloped withouttheintentionofthenetworkoperator. Theimpactofthefirstgenerationofsocialnetworkcuratoralgorithmshasattractedsignificantinterest fromtheresearchcommunity. Perhapsthebestknownhypothesisregardingtheirusagehasbeenthe creationof“filterbubbles”. Inthisphenomenon,usersareexposedtoanincreasinglyrestrictedset ofopinionsandperspectivesbythecurationalgorithmasitover-exploitsitsknowledgebaseabout pre-existinguserpreferencesinordertomaximisetheirengagement[5,6]. Furtherworkhassought toclarifythedecisionstakenbythealgorithms[7]andunderstandtheemotionalresponseofusers toitsapplication[8]. RelatedresearchundertakenbyFacebookhasemphasisedtheimportanceof theindividual’schoiceswhendeterminingtheextenttowhichcurationinfluencesauser’sexposure tochallengingviews[9]. Thesestudiesprovideausefulcontextfortheeffectsproducedbyearly attemptsatsocialnetworkcuration. However,inthisworkweinsteadfocusourattentiononthe potentialconsequencesofthenextgenerationofviablecuratoralgorithms. AnumberofpreviousworkshavealsoexploredthepotentialforformsofArtificialIntelligenceto manipulatehumans,particularlyasaconsequenceofapredictedintelligenceexplosion[10,11],an eventwhichisoftenreferredtoasthesingularity[12]. Themanyrisksofhumanmanipulationby theresultingsuperintelligenceareanalysedindetailin[13]. Previouspredictionsforthetimescale of this event vary, but all consider that if it were to occur, it would require a level of technology thatisnotyetavailable[14,15,13]. Incontrasttothethreatposedbyasuperintelligence,weargue thatthealgorithmicmanipulationofhumansinsocialnetworksisfeasiblewithcurrentlyavailable technology. Morecloselyrelatedtoourwork,thepotentialforpsychologicalparasites(intellectualstimulithat leadtoaddictions)areidentifiedasariskassociatedwiththeimprovingcapabilitiesoftechnologyin [16]. Theserisksareparticularlyabundantinmobilsationsystems—persuasivetechnologiesdesigned to coordinate users towards specific goals [17]. We develop this idea further, arguing that there arespecificrisksposedbythecombinationofcurrentmachinelearningalgorithmsandaccessto abundantuserdatainthesocialnetworkdomain. Settocomeintoforcein2018,theEuropeanGeneralDataProtectionRegulation[18]introducesa rangeofmeasuresofsignificantrelevancetoindustriesheavilyengagedinthecollectionandanalysis ofuserdata. Anyframeworkthatseekstoprovideappropriateregulationforcuratoralgorithmsfaces adauntingtask: itmustseektoprotectthewell-beingofthenetworkparticipantsbutalsostriveto protecttheabilityofthenetworkoperatorstoinnovate. Weconsidertheeffectofthislegislation 2Forsocialnetworkswhosebusinessmodelsarebasedonadvertising,thisobjectivemaybemaximised throughanappropriateproxy,suchasthetotaltimeauserspendseachdayinteractingwiththesite. 2 inthesocialnetworkdomainandassertthatcuratoralgorithmsdeserveparticularattentionfrom regulators. Whenconsideringthepotentialavenuesfortheregulationofcurationalgorithms,itmayproveuseful toconsiderhowotherindustrieshaveapproachedsimilarchallenges. Inrecentyears,regulatorsinthe financialindustryhavebeenfacedwiththetaskofpreventingmarketmanipulationbyincreasingly complicated, algorithmically driven high frequency trading strategies [19]. While some of the proposedregulatoryresponsesarespecifictofinance(forinstance,cancellationtaxeswhichrendera numberofmarketmanipulationstrategiesinfeasible[20]),thefinancialindustryprovidesauseful referencepointforregulatorsinthesocialnetworkdomain(seeSec. 4fordetails). Inthisworkweconsidertherisksandregulationofsocialnetworkcuratoralgorithmsbyformulating theirtaskasareinforcementlearningproblem. Concretely,ourfirstcontributionistodeterminethe risksofanunregulatedsystembyexploringarangeofstrategiesacuratoralgorithmmightemploy withdetrimentaleffectsforusers. Oursecondcontributionistoproposespecificstrategiesforthe saferegulationofcuratoralgorithmsinthecontextofexistingdatalegislationandtoassesstheir potentialeffectivenessinthisrole. 2 EngagementasaLearningProblem Wewillviewtheproblemofmaximisinguserengagementaccordingtosomeutilityfunctionmuch asamachinelearningresearcherworkinginadvertisementmight—asareinforcementlearningtask [21]. Thisframeworkhasbeenshowntobeparticularlyeffectiveinoptimisingcontentselectionfor socialnetworkusers[22]. Atacoarselevel,atypicalreinforcementlearningmodelisbuiltaroundseveralcoreconcepts: • A set of states, S, which fully encode the system and environment we intend to model. Thestateforindividualusersmaybemodelledasanaggregateofthecontentpresented on-screenanda(partiallyobserved)estimateoftheuser’s‘internal’mentalstate. • Asetofpossibleactions,A,whichthesystemcantriggerinreturnfora(possiblydelayed) reward (R). Triggering an action may also cause a state transition. In the examples we consider,anactionmayrepresentcontenttoasocial-networkuser. • Anindicationofreward,utility,orlongtermvalueforthealgorithm(R). Itisagainstthis thattheoperatoradaptsthepolicyfunction,selectingforstrategieswhichmaximisethis reward. • ApolicyfunctionP : S ×A → R(×S). Thismappingessentiallyencodesthestrategy whichthesystempursuesinordertomaximiserewardinthelongtermhorizon. Itishere thatexternalcontrolofthecurationalgorithmmaybeexertedtoavoidpathologicaland potentiallyunethicalbehaviour. Reinforcementlearningannealsonapolicyfunctiontomaximisethevalueandthereforethelong runningutilityofasystem. Itisclearthen,thatitisthiscomponentwhichdeterminesthesophis- ticationoftheuserengagementstrategy,andthereforeitisherethatwefocusourattention. Ata fundamentallevel,thepolicyfunctiondoesnothingmorethanprovideamappingfromthestate-action spacethroughtothescalarvaluefunction. Thesophisticationofthestrategyisthereforestrongly linkedtothecomplexityofthemappingweareabletoexpress, andtodaydeepneuralnetworks areusuallychosenasthesurrogateforthisfunction. Thesearecapableofexpressingverycomplex andnon-intuitivefunctions,asdemonstratedbyGoogle’sAlphaGoproject[23],whereaGoplaying policyfunctionwaslearnedwhichnotonlyoutperformedtophumanplayers,butdidsoviaamixed modeofhuman-likeandhighlynon-intuitivebutoptimalmoves. Otherexamplesofsuchbehaviour arosewhenthesesystemsweretrainedtoplayvideogames. Inparticular,whenGoogletraineda policyforplayingthenotoriousAtariboxergame[24]thesystemlearnedtoexploitweaknessesin thegamedesign,trappingtheopponentinacornerandtherebyguaranteeingvictory. Asresearch continues,wecanenvisageaworldinwhichtheseapproachesareeffectivelybroughttobearonthe “game”ofmaximisingnetworkprofitability. Ifgovernedsolelybythisutilityfunction,wesuggest thatequivalentpathologiesinhumanbehaviourmaybediscoveredandexploited. 3 3 ManipulationThroughCuration Inthissectionwediscusstherangeofmanipulationstrategiesavailabletoacuratoralgorithmseeking tooptimisetheprofitabilityofasocialnetwork. Inthiscontext,wetakemanipulationtomeantheart ofdeliberatelyinfluencingaperson’sbehaviourtobenefitsomeobjective. Webeginbydescribing theformsofmanipulationthatareapplicableinthedomainofsocialnetworks. Wethenintroducea simplecategorisationofthedifferentformsofmanipulationandofferexamplesofthestrategiesa curatoralgorithmmightdevelopwithdetrimentaleffectsforitsusers. Manipulationformsanaturalcomponentofhumaninteractionandcantakemanyforms,ranging from direct requests to subtle and intentionally hidden signals. A number of previous studies have demonstrated how human behaviour can be influenced with subtle visual and verbal clues [25, 26, 27, 28]. Research intothe designof sitefeatures hasfurther demonstratedthe abilityof operatorsto“steer”userbehaviourin[29]. Ofparticularrelevancetothiswork,ithasbeenshown thattheemotionalstatesofsocialnetworkuserscanbeinfluencedbyselectivelyfilteringthecontent producedbytheirfriends[30]. Influential early work in the field of behavioural psychology determined that animals could be manipulatedmosteffectivelyiftheyarerewardedonavariable,unpredictableschedule[31]. This behaviourhasbeenusedprofitablybycasinoswhooffergamblerssurpriserewardstokeepthem hookedtotheactioninthemidstofalosingstreak[32]. Similarideashavebeenappliedtogame designtokeepplayersengagedforlongerbyunpredictablyvaryingthedurationofin-gametasks [33]. Thesepsychologicaltraitsexemplifythekindofin-builtbehavioursthatcouldbediscovered andexploitedbyacuratoralgorithm. Inordertoexplorethespecificformsofstrategyavailabletoacuratoralgorithmweproposeasimple categorisationofmanipulation. Wedefineamanipulationtobeoffirstorderifthemanipulationis directandtheobjectiveofthemanipulatoristransparenttotheparticipant. Amanipulationisdefined tobeofsecondorderifitisindirect,buttheobjectiveremainstransparenttotheparticipant. Further, weconsideramanipulationtobeofthirdorderifitisindirectandthemeansbywhichtheobjective isattainedarenottransparenttotheparticipant3. Thesecategoriesmaybeillustratedwithasimpleexample. Considerabarownerwishingtoincrease drinkssalesattheirestablishment. Eachevening,theownermaychoosetosimplyaskcustomers directlytopurchasemoredrinks. Thisstrategy,correspondingtoafirstordermanipulation,hasthe benefit of simplicity but may not lead to optimal drinks sales (or indeed the renewal of their bar licence). Theownermayinsteadaimtoincreasesaleswithadvertisementsillustratingtheenjoyment ofothercustomersastheyrefreshthemselveswithdrinksfromthebar. Thisformofadvertisingaims toevokeasenseofdesireinthecustomerswhichmayleadindirectlytothepurchaseofmoredrinks. However,theobjectiveoftheadvertremainstransparenttothecustomer,correspondingtoasecond ordermanipulation. Finally,ashrewdbarownermayemployathirdstrategy,inwhichtheyprovide freesnackstocustomersofthebar. Thesnacks,however,areheavilysalted,andafterconsuming themthecustomersfindtheirthroatsparchedandinneedofimmediaterefreshment. Thisstrategyis bothindirectandnottransparenttoallbuttheexperiencedcustomers,correspondingtoathirdorder manipulation. Weremarkherethataccesstodetailedinformationaboutthetargetplaysanimportantroleinthe ability to manipulate them. Should an unethical bar owner overhear sensitive information about thepersonallifeofacustomeratthebartheyhavethepotentialtopursuefurtherstrategies,such asensuringthecustomerlosestheirjobsothattheyaremorelikelytospendtimedrinkingatthe establishment. Wemightassumethatacuratoralgorithmseekingtomaximiseprofitabilitywillnaturallyexplore first and second order manipulations as it seeks to advertise products to its user base. Aided by accesstodetaileduserinformation,itcanmakepowerfulinferencesaboutwhichinformationshould be displayed at each instant. Consider, for example, the marketing of an energy drink. With the knowledgethatauserisastudent,thattheyareawakebeyondtheirusualsleepcycle,thatthedateof theirexamsisdrawingnearandthattheironlineactivityshowsindicationsoffatigue,thecuratorcan 3Notethatthedistinctionbetweenthesecategoriesrestsonthedifficultassessmentofthecognitiveabilities ofthetarget[34].Themanipulatormaydeterminethattheintentionofagivensetofbehavioursistransparentto asophisticatedtarget,butnottoasimpletarget. 4 selectanoptimaltimeandcontextforthedisplayofanadvert. Nowimagineamoresophisticated algorithmcapableofpursuingthirdordermanipulations. Suchanalgorithmmightchoosetodisplay contentwhichhadbeenselectedwiththespecificgoalofexhaustingtheuser. Thiscouldbeachieved bytriggeringpredictablerepeatbehavioursgleanedfromanin-depthknowledgeoftheirbrowsing habits. Indeed, over longer time horizons, the curator might determine that an effective method for increasing the sales of energy drinks is the distortion of the user’s sleeping patterns. To take anotherexample, consider acuratoralgorithmseeking touseinformationabout socialgroupsto increasesalesofdatingsitememberships.Whilesimplemanipulationscouldleadittopresentcontent encouraging individual users to search for partners, it could pursue third order manipulations by intentionallyencouragingsubsetsofsocialgroupstocommunicateinamannerthatexcludesother members,activelyevokingafeelingoflonelinessintheaffectedpartytoincreasetheirresponsiveness toadvertising. A recent example of this strategy exploration principle in action can be found in the efforts of a collectionofcompaniesseekingtooptimiseadvertisingrevenueduringtheUnitedStatespresidential electionin2016[35]. Throughsimpletrialanderror,theydeterminedthatcarefullytargetedfake politicalnewsstorieswereextremelyeffectiveinmaximisingclick-throughs. Sincethisstrategywas optimisingtheirobjective,theydoubleddownonthisapproachandproducedasmuchcontentas possiblewithoutregardforitseffectontheusersofthenetwork. Withthesameobjective,evena comparativelysimplecuratoralgorithmwouldbecapableofdevelopingthisstrategy. We note that it is certainly not the case that all strategies pursued by a curator algorithm will be detrimental for users. Indeed, the energy drinks may give the tired student the boost required to raisetheirgrade,whilethepreviouslylonelyusermayfindhappinessthroughtheirnewdatingsite membership. However,perhapsthemoststrikingaspectoftheAtarigame-playingalgorithm[24] wasnotthatitwascapableofsurpassinghumanperformance,butratherthatitcameupwith“cheat” strategiesthathumanplayershadnotpreviouslyconsidered(e.g. theboxerstrategydescribedin Sec.2). Similarly,althoughthemanipulationexamplesdescribedabovearesimpleandinterpretable, wesuggestthatthecuratoralgorithmiscapableofdevelopingsophisticated,uninterpretablestrategies for manipulating users as they optimise their objective. By their very nature, such strategies are difficult to predict and therefore difficult to regulate. It is however an issue that is worthy of considerationifwewishtoavoidthediscoveryofsimilar“cheat”strategiesforhumanmanipulation. 4 CuratorRegulation Dosocialnetworkcuratoralgorithmsdeservespecialattentionfromregulators? InSec. 1,weargued thattherisksofhigherordermanipulationsresultfromprovidingcuratoralgorithmswiththreekey assets: extensiveaccesstouserdata,theabilitytodevisesophisticatedstrategies(potentiallybeyond theunderstandingofhumanoperators)andaneffectivemechanismforevaluatingtheeffectsofits strategies. Inthissection, weassesstheneedforregulationinsocialnetworkcuratoralgorithms inthecontextofthesethreeareas. WebeginbydiscussingtheGeneralDataProtectionRegulation (GDPR)recentlyintroducedbytheEuropeanUnionanditsimplicationsforthestrategiesavailableto algorithmsoperatinginthesocialnetworkdomain. Specifically,weexamineitsabilitytosafeguard usersfromhigherordermanipulationsthroughitsrequirementofalgorithminterpretability. Next,we explorestrategiesforthespecificregulationofreinforcementlearning-basedcuratoralgorithmsand makerecommendationsfortheirapplication. Finally,wediscussthechallengesfacedbyregulators operatinginindustriesinwhichalgorithminterpretabilityisofteninfeasibleasausefulreferencefor regulatorsconsideringtheproblemofcuratormanipulation. Asmodernsocialnetworksdevelopaglobaluserbase,theybecomesubjecttoadiverserangeof nationaldataandprivacyregulations[36],aswellaslawsgoverningthetransborderdataflowsthat occurintheoperationofainternationalorganisation[37]. Ofthese,onesetofregulationswhich holdsparticularsignificancefortheoperationofcuratoralgorithmsistheGeneralDataProtection Regulation,settocomeintoforceacrosstheEuropeanUnionin2018[18]. Amongrulesgoverning thestorageandusageofpersonaldatawhichwillapplyinsocialnetworkdomain,itscreationofa socalled“righttoexplanation”[38]hassignificantconsequencesforthedesignofalgorithmsthat operateonpersonaldata. Byrequiringthatcompaniesperformingautomateddecisionmakingbased onpersonaldatamustbecapableofsupplying“meaningfulinformationaboutthelogicinvolved”, theregulationplacesheavyemphasisonalgorithminterpretability. 5 Algorithminterpretabilityhasbeenalongstandingtopicofinterestinmachinelearning,yielding techniquesthatmodifyandextendmodelstoexplaintheirdecisions[39,40]alongsideeffortsto improvenaturallyinterpretablealgorithmstomakethemcompetitivewiththeiropaquecounterparts [41]. However,whiletherehasbeenagreatdealofresearchinterestinimprovingtheunderstanding ofdeepneuralnetworks(usingtechniquessuchasrandomperturbation[42], invarianceanalysis [43,44,45],visualisation[46,47,48]anddimensionalityreduction[49]),theinterpretationofthese models remains notoriously challenging. Consequently, it is not clear whether these models are currentlycapableofprovidingthe“meaningfulinformation”requiredbytheregulation. Toachievecompliance,curatoralgorithmsmaythereforeberestrictedtoasetofsimplefunction classes. Asaresult,thepotentialsophisticationofthepolicyfunctiondescribedinSec. 2wouldbe curtailedandhigherordermanipulationstrategieswouldbeunlikelytoarise. However,wenotethat therearetworeasonswhythisregulationmaynotbeaneffectivesafeguardforsocialnetworkusers. Firstly,theregulationsetssuchcomprehensiverequirementsthatitmaybecomemeaninglessinthe lawbooks[50]. Secondly,alackofconsensusonpreciselywhatitmeansforamodeltopossessthe propertyofinterpretabilityorbecapableofproviding“meaningfulinformation”makesitextremely difficulttoassesstheformsofalgorithmthatwouldbecompliantwiththeregulation. Indeed,under certaincriteriaithasbeenobservedthatdeepneuralnetworksmaybeconsiderednolessinterpretable thanlinearmodels[51]. Ineithercase,iftheambiguitiesoftherequirementofinterpretabilityshouldrenderitineffectivein preventinghigherordermanipulations,whatoptionsremainavailabletoregulators? Itmaybethat evenwithoutcomprehensivemodelunderstanding,reasonableguaranteesabouttheactionstaken bythemodelcanbeachieved. Inanumberofcomplexindustrialcontrolsystems,apolicyfunction existsimplicitlythroughafunctionalapproximationtothephysicsofthesystem,ratherthansolely throughthedirectinferenceofsystemdynamicsfromdata. Asanexample,standardcommercial autopilotsrelyonanimplicitpolicyfunctionthroughsophisticatedcontrolsystems[52,53]andare requiredtoprovidereasonableguaranteesabouttheirbehaviourtoavoidundesirableoutcomesforthe operator. Toachievethis,controllerdesignerschooseapproximationstothesystemdynamicsinorder toarriveatanimplicitpolicywhichisguaranteedtoavoidunfavourableregionsofstate/actionspace. Thereareavarietyofsubclassesofapproximationwhichleadtoprovably“correct”systems(see [54]foranexample). Whileitremainschallengingtoprovideguaranteesonthelongtermbehaviour ofhighlycomplexpolicyfunctionslearnedfromdata,thereisgrowinginterestinachievingaccurate credibleintervalestimationsfortheoutputsproducedbydeepneuralnetworks[55]. Researchin thisareamayprovidesomeempiricalunderstandingofthebehaviourwemightexpectfromagiven policyfunctionasitadaptstonewobservations. Othermethodshavedemonstratedthepotential of combining a series of locally simple models [56], an approach that has the potential to admit moreaccessibleanalysis. Iffurtherworkisabletoprovideappropriatestate-actionspacebehaviour guarantees,itshouldbeabletorestrictmanipulativebehaviourwithoutarequirementonthelow levelinterpretabilityofthepolicyfunction. Aninterestingalternativefortheregulationofalgorithmsthatliebeyondhumaninterpretationcanbe foundinthegrowingfieldofmachineethics[57,58]. Asthefieldofmachinelearningcontinuesto develop,itmayfrequentlybethecasethatthemostusefulalgorithmsdonotreadilyadmithuman interpretation. Ratherthanprohibitingtheuseofthesealgorithms,itmaybepossibleforregulators topreventmanipulationbyrequiringthatcuratoralgorithmsactinamannerthatisconsistentwith acarefullyspecifiedsetofethicalchoices. However,wenotethatatpresentthisapproachfacesa numberofchallengesthatmaketheimplementationofanethicalcuratoralgorithmextremelydifficult [59]. While provable behavioural guarantees and machine ethics could prove to be effective tools for regulatorsinthelongterm,intheshorttermwesuggestthatamorepragmaticapproachmaybe required. Asnotedabove,thethirdkeyfactorenablingcuratoralgorithmstodevelopmanipulative behaviouristheavailabilityofanimmediateandeffectivefeedbackmechanismforevaluatingthe responseofusers. Wethereforesuggestthatapracticalshorttermsolutioncanbeachievedthrough theconstructionapartialfirewallrestrictingtheflowofinformationthatprovidesthismechanism in social networks. However, if applied without careful consideration, this method runs the risk ofplacinganunnecessarilystrongconstraintontheabilityofsocialnetworkoperatorstoimprove theircurationservicefortheirusers. Regulatorsmustthereforeseekanappropriatebalancebetween safeguardingusersfromtherisksofmanipulationandenablingoperatorstoinnovateandproduce productswhichwillbenefitthoseusers. 6 Asabriefaside,wenotethatherethatalthoughthefinancialindustrydiffersinmanywaysfrom the world of social networking, it provides a useful reference for the difficult challenges facing regulatorsinthesocialnetworkdomain. Specifically,regulatorsseektopreventmarketmanipulation, apracticebywhichparticipantsartificiallydistortinformationtotheirbenefit[60]. However,while manipulationthroughhuman-basedtradingpracticeshavelongbeenoutlawed,legislatorsarestill exploringapproachestoregulatingtheHighFrequencyTrading(HFT)algorithmsthatincreasingly dominatethemarketplace. Byoperatingatspeedshumanscannotmatchthesealgorithmsareableto manipulate4 themarketinwaysthataredifficulttodetect[61,62]. Theneedfortheregulationof thesealgorithmswasbroughtintosharpreliefbytheirroleinthe“FlashCrash”ofthestockmarket in2010,aneventwhichresulteda9%indexdropinasinglehouroftrading[63]. Oneregulatory approachtopreventingalgorithmicmarketmanipulationhasbeentheintroductionalgorithmtagging, aprocessinwhichtradersmustprovidetheidentityofthealgorithmresponsibleforatrade[64]. While this approach has been helpful in improving regulators’ understanding of the interactions betweendifferentmarketparticipants,ithasnotyetbeendemonstratedtobeeffectiveinpreventing manipulation[65]. Attimes,regulatorshavetakenthemoredirectapproachofrequestingaccessto thealgorithmsthemselves[66],butwhenalgorithminterpretabilityisnotfeasiblethisactionisof limitedvalue. Inshort,despiteextensiveexperienceinregulatingtradingpracticestopreventmarket manipulation,financialindustryregulatorshaveyettoachieveaunifiedapproachtotheproblemof algorithmicmanipulation. Thisshouldserveasawarningthataregulatorysolutiontothepotentially morecomplicatedissueofpreventinghumanmanipulationmayproveextremelychallenging. In summary, the task facing regulators seeking to prevent the manipulation of users by social networkcuratoralgorithmsisadifficultone. TheboldapproachtakingbytheEuropeanUnionmay prove effective in combatting this issue, but it remains to be seen whether setting a requirement ofinterpretabilityisbothpracticalandenforceable. Shouldthisbethecase,weproposeasimple firewall-basedapproachasshort-termsafeguardforusersuntilmoresophisticatedtechniquescanbe developedtopreventtherisksofmanipulation. 5 Conclusions Machinelearningapplicationshavethepotentialtohaveanenormouslypositiveimpactonawide rangeofindustriesandonthedailylivesofpeoplearoundtheglobe.However,whileitisencouraging to see the research and development of the algorithms driving these applications making rapid progress,itisimportanttonotethatwhenprovidedwithaccesstoabundantquantitiesofpersonal data,thesealgorithmsalsopresentrisks. Inthisworkwediscussonesuchrisk,namelythepotential forthemanipulationofusersbythecuratoralgorithmofasocialnetwork. Toclarifythedangers associatedwiththispossibility,wehighlightedstrategieswebelievecouldbefeasiblydiscovered throughcurrentreinforcementlearningtechniquesgivenadequateaccesstoavailablestoreduser data. Regulators are faced with a difficult task as they try to allow new technologies to flourish whileprotectingtheusersofsocialnetworksfromformsofmanipulationthatareinherentlydifficult to detect. In an effort to address this issue, we assessed potential avenues for regulating curator algorithmsandofferedrecommendationsfortheiruse. Asmachinelearningcontinuestoprogress, weexpecttoseesimilartradeoffsbetweenopportunitiesandrisksemergeinacrossotherindustries whichrelyheavilyonpersonaldata. 5.1 Acknowledgements The authors would like to thank Thomas Smith, Hannah Hjerpe-Schroeder, Ruth Fong, Ankush Gupta,JamesThewlis,Anna-ElizabethShakespeare,JudithAlbanieandStephenRobertsforhelpful discussionsandNeilLawrence,whosetalksinspiredmuchofthiswork. SamuelAlbanieandHillary ShakespearearefundedbytheESPRCEP/L015897/1(AIMSCDT)grant. TomGunterissupported byUKResearchCouncils. 4Onesuchtechniqueisspoofing,apracticewhichinvolvesplacinglargesellordersabovethecurrentasking pricewhicharequicklycancelledifthepricebeginstorise. 7 References [1] WLanceBennettandAlexandraSegerberg. 3thelogicofconnectiveaction: Digitalmediaandthe personalizationofcontentiouspolitics. CivicMedia:Technology,Design,Practice,page77,2016. [2] JulianBühler,AaronWBaur,MarkusBick,andJiminShi. Bigdata,bigopportunities:Revenuesources ofsocialmediaservicesbesidesadvertising. InConferenceone-Business,e-Servicesande-Society,pages 183–199.Springer,2015. [3] MichelBallingsandDirkVandenPoel. Crminsocialmedia: Predictingincreasesinfacebookusage frequency. EuropeanJournalofOperationalResearch,244(1):248–260,2015. [4] SophiaBLiu. Trendsindistributedcuratorialtechnologytomanagedatadelugeinanetworkedworld. Upgrade:TheEuropeanJournalfortheInformaticsProfessional,11(4):18–24,2010. [5] EliPariser. Thefilterbubble:WhattheInternetishidingfromyou. PenguinUK,2011. [6] DavidLazer. Theriseofthesocialalgorithm. Science,348(6239):1090–1091,2015. [7] MotahhareEslami,AmirhosseinAleyasen,KarrieKarahalios,KevinHamilton,andChristianSandvig. Feedvis:Apathforexploringnewsfeedcurationalgorithms. InProceedingsofthe18thACMConference CompaniononComputerSupportedCooperativeWork&SocialComputing,pages65–68.ACM,2015. [8] Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, KevinHamilton, andChristianSandvig. Ialwaysassumedthatiwasn’treallythatclose to[her]: Reasoningaboutinvisiblealgorithmsinnewsfeeds. InProceedingsofthe33rdAnnualACM ConferenceonHumanFactorsinComputingSystems,pages153–162.ACM,2015. [9] EytanBakshy,SolomonMessing,andLadaAAdamic.Exposuretoideologicallydiversenewsandopinion onfacebook. Science,348(6239):1130–1132,2015. [10] IrvingJohnGood. Speculationsconcerningthefirstultraintelligentmachine. Advancesincomputers, 6:31–88,1966. [11] VernorVinge. Thecomingtechnologicalsingularity:Howtosurviveinthepost-humanera. 1993. [12] StanislawUlam. Tributetojohnvonneumann.64,#3,part2. BulletinoftheAmericanMathematical Society.p5. [13] NickBostrom. Superintelligence:Paths,dangers. Strategies,pages30–36,2014. [14] NickBostrom. Howlongbeforesuperintelligence? 1998. [15] RayKurzweil. Thesingularityisnear:Whenhumanstranscendbiology. Penguin,2005. [16] FrancisHeylighen. Returntoeden?promisesandperilsontheroadtoaglobalsuperintelligence. TheEnd oftheBeginning:Life,SocietyandEconomyontheBrinkoftheSingularity.Retrievedfromhttp://pespmc1. vub.ac.be/Papers/BrinkofSingularity.pdf,2014. [17] FrancisHeylighen,IavorKostov,andMixelKiemen. Mobilizationsystems:technologiesformotivating andcoordinatinghumanaction. TheNewDevelopmentParadigm:Education,KnowledgeEconomyand DigitalFutures.Routledge.Retrievedfromhttp://pcp.vub.ac.be/Papers/MobilizationSystems.pdf,2013. [18] EuropeanCouncil. Generaldataprotectionregulation,2016. http://www.eugdpr.org. [19] Peter Gomber and Markus Gsell. Catching up with technology-the impact of regulatory changes on ecns/mtfsandthetradingvenuelandscapeineurope. Competition&Reg.NetworkIndus.,7:535,2006. [20] MattPrewitt. High-frequencytrading: Shouldregulatorsdomore. Mich.Telecomm.&Tech.L.Rev., 19:131,2012. [21] RichardSSuttonandAndrewGBarto. Reinforcementlearning:Anintroduction,volume1. MITpress Cambridge,1998. [22] LihongLi,WeiChu,JohnLangford,andRobertESchapire. Acontextual-banditapproachtopersonalized newsarticlerecommendation. InProceedingsofthe19thinternationalconferenceonWorldwideweb, pages661–670.ACM,2010. [23] DavidSilver,AjaHuang,ChrisJMaddison,ArthurGuez,LaurentSifre,GeorgeVanDenDriessche,Julian Schrittwieser,IoannisAntonoglou,VedaPanneershelvam,MarcLanctot,etal. Masteringthegameofgo withdeepneuralnetworksandtreesearch. Nature,529(7587):484–489,2016. 8 [24] VolodymyrMnih,KorayKavukcuoglu,DavidSilver,AlexGraves,IoannisAntonoglou,DaanWierstra, andMartinRiedmiller. Playingatariwithdeepreinforcementlearning. arXivpreprintarXiv:1312.5602, 2013. [25] MaxErnest-Jones,DanielNettle,andMelissaBateson. Effectsofeyeimagesoneverydaycooperative behavior:afieldexperiment. EvolutionandHumanBehavior,32(3):172–178,2011. [26] StewartKettle,MarcoHernandez,SimonRuda,MichaelSanders,etal. Behavioralinterventionsintax compliance:evidencefromguatemala. Technicalreport,TheWorldBank,2016. [27] MichaelLaakasuo,JussiPalomäki,andMikkoSalmela. Emotionalandsocialfactorsinfluencepoker decisionmakingaccuracy. JournalofGamblingStudies,31(3):933–947,2015. [28] ThomasCLeonard. Richardh.thaler,cassr.sunstein,nudge:Improvingdecisionsabouthealth,wealth, andhappiness. ConstitutionalPoliticalEconomy,19(4):356–360,2008. [29] AshtonAnderson,DanielHuttenlocher,JonKleinberg,andJureLeskovec. Steeringuserbehaviorwith badges. InProceedingsofthe22ndinternationalconferenceonWorldWideWeb,pages95–106.ACM, 2013. [30] AdamDIKramer,JamieEGuillory,andJeffreyTHancock. Experimentalevidenceofmassive-scaleemo- tionalcontagionthroughsocialnetworks.ProceedingsoftheNationalAcademyofSciences,111(24):8788– 8790,2014. [31] BFSkinner. Thebehavioroforganisms:anexperimentalanalysis. 1938. [32] ChristinaBinkley. Takingretailers’cues,harrah’stapsintoscienceofgambling. TheWallStreetJournal, pageA1,2004. [33] JoséPZagal,StaffanBjörk,andChrisLewis. Darkpatternsinthedesignofgames. 2013. [34] WakoYoshida,RayJDolan,andKarlJFriston.Gametheoryofmind.PLoSComputBiol,4(12):e1000254, 2008. [35] Dan Tynan. How facebook powers money machines for obscure political ’news’ sites, August 2016. https://www.theguardian.com/technology/2016/aug/24/ facebook-clickbait-political-news-sites-us-election-trump. [36] LeeABygrave. Dataprivacylaw:Aninternationalperspective. OxfordUniversityPressOxford,2014. [37] ChristopherBarthKuner. Transborderdataflowsanddataprivacylaw. OxfordUniversityPress,2013. [38] Bryce Goodman and Seth Flaxman. Eu regulations on algorithmic decision-making and a “right to explanation”. InICMLWorkshoponHumanInterpretabilityinMachineLearning(WHI2016),2016. [39] WilliamRSwartout. Xplain:Asystemforcreatingandexplainingexpertconsultingprograms. Artificial intelligence,21(3):285–325,1983. [40] GregRidgeway,DavidMadigan,ThomasRichardson,andJohnO’Kane.Interpretableboostednaïvebayes classification. InKDD,pages101–104,1998. [41] YinLou,RichCaruana,andJohannesGehrke. Intelligiblemodelsforclassificationandregression. In Proceedingsofthe18thACMSIGKDDinternationalconferenceonKnowledgediscoveryanddatamining, pages150–158.ACM,2012. [42] JulianDOldenandDonaldAJackson. Illuminatingthe“blackbox”: arandomizationapproachfor understandingvariablecontributionsinartificialneuralnetworks. Ecologicalmodelling,154(1):135–150, 2002. [43] IanGoodfellow,HonglakLee,QuocVLe,AndrewSaxe,andAndrewYNg. Measuringinvariancesin deepnetworks. InAdvancesinneuralinformationprocessingsystems,pages646–654,2009. [44] JoanBrunaandStéphaneMallat. Invariantscatteringconvolutionnetworks. IEEEtransactionsonpattern analysisandmachineintelligence,35(8):1872–1886,2013. [45] K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. InCVPR,2015. OralPrensetation. [46] KarenSimonyan,AndreaVedaldi,andAndrewZisserman.Deepinsideconvolutionalnetworks:Visualising imageclassificationmodelsandsaliencymaps. arXivpreprintarXiv:1312.6034,2013. 9 [47] MatthewDZeilerandRobFergus. Visualizingandunderstandingconvolutionalnetworks. InEuropean ConferenceonComputerVision,pages818–833.Springer,2014. [48] AravindhMahendranandAndreaVedaldi. Understandingdeepimagerepresentationsbyinvertingthem. In2015IEEEconferenceoncomputervisionandpatternrecognition(CVPR),pages5188–5196.IEEE, 2015. [49] AlfredoVellido,JoséDavidMartín-Guerrero,andPauloJGLisboa. Makingmachinelearningmodels interpretable. InESANN,volume12,pages163–172.Citeseer,2012. [50] Bert-Jaap Koops. The trouble with european data protection law. International Data Privacy Law, 4(4):250–261,2014. [51] Zachary C Lipton, David C Kale, Charles Elkan, Randall Wetzell, Sharad Vikram, Julian McAuley, RandallCWetzell,ZhanglongJi,BalakrishnanNarayaswamy,Cheng-IWang,etal. Themythosofmodel interpretability. IEEESpectrum,2016. [52] S.R.BarrosdosSantosandN.M.F.deOliveira. Longitudinalautopilotcontrollerstestplatformhardware intheloop. In2011IEEEInternationalSystemsConference,pages379–386,April2011. [53] PierreApkarian,PascalGahinet,andGregBecker. Self-scheduledh-infinitycontroloflinearparameter- varyingsystems:adesignexample. Automatica,31(9):1251–1261,1995. [54] AlbertoIsidori. NonlinearControlSystems. Springer-VerlagNewYork,Inc.,Secaucus,NJ,USA,3rd edition,1995. [55] YarinGalandZoubinGhahramani.DropoutasaBayesianapproximation:Representingmodeluncertainty indeeplearning. InProceedingsofthe33rdInternationalConferenceonMachineLearning(ICML-16), 2016. [56] SergeyLevineandPieterAbbeel. Learningneuralnetworkpolicieswithguidedpolicysearchunder unknowndynamics. InNeuralInformationProcessingSystems(NIPS),2014. [57] James H Moor. The nature, importance, and difficulty of machine ethics. IEEE intelligent systems, 21(4):18–21,2006. [58] WendellWallachandColinAllen. Moralmachines:Teachingrobotsrightfromwrong. OxfordUniversity Press,2008. [59] MilesBrundage. Limitationsandrisksofmachineethics. JournalofExperimental&TheoreticalArtificial Intelligence,26(3):355–372,2014. [60] RobertAJarrow. Marketmanipulation,bubbles,corners,andshortsqueezes. Journaloffinancialand QuantitativeAnalysis,27(03):311–336,1992. [61] JayaramMuthuswamy,JohnPalmer,NivineRichie,andRobertWebb. High-frequencytrading:implica- tionsformarkets,regulators,andefficiency. TheJournalofTrading,6(1):87–97,2011. [62] MarcLenglet. Conflictingcodesandcodingshowalgorithmictradingisreshapingfinancialregulation. Theory,Culture&Society,28(6):44–66,2011. [63] USSecurities,ExchangeCommission,CommodityFuturesTradingCommission,etal. Findingsregarding themarketeventsofmay6,2010. WashingtonDC,2010. [64] Hessisches Ministerium. Guidelines to the adherence to the requirement of the labelling of trading algorithms,2014. [65] NathanCoombs. Whatisanalgorithm?financialregulationintheeraofhigh-frequencytrading. Economy andSociety,45(2):278–302,2016. [66] Sarah N. Lynch and J. Spicer. Regulators seek trading secrets, August 2011. Available at http://www.reuters.com/article/us-financial-regulation-algos-idUSTRE7806J420110901. 10