Table Of ContentSee discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/318820255
Algorithmic regulation: A critical interrogation
Article in Regulation & Governance · July 2017
DOI: 10.1111/rego.12158
CITATIONS READS
17 1,268
1 author:
Karen Yeung
University of Birmingham
32 PUBLICATIONS 513 CITATIONS
SEE PROFILE
All content following this page was uploaded by Karen Yeung on 09 February 2018.
The user has requested enhancement of the downloaded file.
(Yeung Algorithmic Regulation 2017 accepted.doc)
The published version of this paper will appear in a forthcoming issue of Regulation &
Governance available at http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1748-5991
Algorithmic Regulation: A Critical Interrogation
Karen Yeung
Professor of Law, Centre for Technology, Ethics, Law & Society (TELOS)
King’s College London
This paper has been accepted for publication by Regulation & Governance. Please do not
distribute without the author’s permission.
Abstract:
Innovations in networked digital communications technologies, including the rise of ‘Big
Data’, ubiquitous computing and cloud storage systems, may be giving rise to a new
system of social ordering known as algorithmic regulation. Algorithmic regulation refers
to decision-making systems that regulate a domain of activity in order to manage risk or
alter behaviour through continual computational generation of knowledge by
systematically collecting data (in real time on a continuous basis) emitted directly from
numerous dynamic components pertaining to the regulated environment in order to
identify and, if necessary, automatically refine (or prompt refinement of) the system’s
operations to attain a pre-specified goal.
It provides a descriptive analysis of algorithmic regulation, classifying these decision-
making systems as either reactive or pre-emptive, and offers a taxonomy that identifies 8
different forms of algorithmic regulation based on their configuration at each of the
three stages of the cybernetic process: notably, at the level of standard setting (adaptive
vs fixed behavioural standards); information-gathering and monitoring (historic data vs
predictions based on inferred data) and at the level of sanction and behavioural change
(automatic execution vs recommender systems). It maps the contours of several
emerging debates surrounding algorithmic regulation, drawing upon insights from
regulatory governance studies, legal critiques , surveillance studies and critical data
studies to highlight various concerns about the legitimacy of algorithmic regulation
Keywords: big data, algorithms, surveillance, enforcement, automation
1
Algorithmic Regulation : A Critical Examination
1. Introduction
“It’s time for government to enter the age of big data. Algorithmic regulation is an
idea whose time has come” (Tim O’Reilly, CEO of O’Reilly Media Inc).
A so-called Big Data revolution is currently underway, which many claim will prove as
disruptive to society in the 21st century as Henry Ford’s system of mass production in the
late 19th century (boyd and Crawford 2012). Although ‘Big Data’ has been variously
defined, I use the term to refer to the socio-technical ensemble which utilises a
methodological technique that combines a technology (constituted by a configuration of
information-processing hardware and software that can sift and sort vast quantities of
data in very short times) with a process (through which algorithmic processes are applied
to mine a large volume of digital data to find patterns and correlations within that data,
distilling the patterns into predictive analytics, and applying the analytics to new data)
(Cohen 2012: 1919). The excitement surrounding Big Data is rooted in its capacity to
identify patterns and correlations that could not be detected by human cognition,
converting massive volumes of data (often in unstructured form) into a particular, highly
data-intensive form of knowledge, and thus creating a new mode of knowledge
production (Cohen 2012: 1919).
Industries, academics and governments are enthusiastically embracing these
technologies, all seeking to harness their tremendous potential to enhance the quality
and efficiency of many activities, including the task of regulation, which this paper
interrogates by critically examining the phenomenon of ‘algorithmic regulation’. It draws
upon selective insights from legal and social scientific literature, highlighting emerging
critiques of algorithmic power and the rise of automated data-driven systems to inform
decision-making and regulate behaviour. My primary aim is to map the contours of
emerging debates, raising questions for further research rather offering definitive
answers, proceeding in four parts. Part I offers a working definition of algorithmic
regulation. Part II then constructs a taxonomy of algorithmic regulatory systems based
on their configuration at each of the three stages of the cybernetic process: notably, at
the level of standard setting (adaptive vs simple, fixed behavioural standards);
information-gathering and monitoring (historic data vs predictions based on inferred
2
data), and at the level of sanction and behavioural change (automatic execution vs
recommender systems). Parts III, IV and V provide a critical analysis of algorithmic
regulation, identifying concerns about its legitimacy drawn selectively from several
strands of academic literature, including regulatory governance and public
administration, legal scholarship, surveillance studies and critical data studies. Part VI
concludes, sketching the contours of a broad research agenda anchored within legal and
regulatory governance scholarship.
2. The mechanisms and forms of algorithmic regulation
2.1 What is algorithmic regulation?
Although Silicon Valley entrepreneur Tim O’Reilly exhorts governments to embrace
algorithmic regulation to solve policy problems he does not define algorithmic regulation,
but merely points to various technological systems1 which he claims share four features:
(a) a deep understanding of the desired outcome; (b) real-time measurement to
determine if that outcome is being achieved; (c) algorithms (i.e. a set of rules) that make
adjustments based on new data; and (d) periodic, deeper analysis of whether the
algorithms themselves are correct and performing as expected (O’Reilly 2013). Because
greater precision and rigour is required for critical analysis, I begin by offering a definition
of algorithmic regulation, exploring what it means to describe something as ‘algorithmic’,
and then explaining how I will understand the term ‘regulation’.
In their broadest sense, algorithms are encoded procedures for solving a problem by
transforming input data into a desired output (Gillespie 2013; 2014). Although algorithms
need not be implemented in software, computers are fundamentally algorithm machines,
designed to store and read data, apply mathematical procedures to data in a controlled
fashion, and offer new information as the output. Even when confined to software, the
term ‘algorithm’ may be variously understood. Software engineers are likely to adopt a
technical understanding of algorithms , referring to the logical series of steps for
organising and acting on a body of data to achieve a desired outcome quickly which
1 He refers to motor vehicle fuel emissions systems, airline automatic pilot systems, credit
card fraud detection systems, drug dosage monitoring by medical professionals, internet spam
filters and general internet search engines: O’Reilly (2013).
3
occurs after the generation of a ‘model’, i.e. the formalisation of the problem and the
goal in computational terms (Gillespie 2013; 2014; Dourish 2016). But social scientists
typically use the term as an adjective to describe the sociotechnical assemblage which
includes, not just algorithms, but also the computational networks in which they
function, the people who design and operate them, the data (and users) on which they
act, and the institutions that provide these services, all connected to a broader social
endeavour and constituting part of a family of authoritative systems for knowledge
production. Accordingly, Gillespie suggests that, when describing something as
‘algorithmic’, our concern is with the insertion of procedure that is produced by, or
related to, a socio-technical information system that is intended by its designers to be
functionally and ideologically committed to the computational generation of knowledge.
For him, ‘what is central is the commitment to procedure, and the way procedure
distances its human operators from both the point of contact with others and the mantle
of responsibility for the intervention they make’ (Gillespie 2014).
Although computational algorithms include those which encode simple mathematical
functions, the excitement surrounding Big Data is largely attributable to sophisticated
machine learning algorithms , fed by massive (and often unstructured) data sets, that
operate computationally and depart from traditional techniques of statistical modelling
(Dourish 2016: 7). Traditional statistical modelling requires the analyst to specify a
mathematical function containing selected explanatory variables and, through regression
analysis, enables the identification of the goodness of fit between the data and these
analytic choices. In contrast, machine learning does not require a priori specification of
functional relationships between variables. Rather, the algorithms operate by mining the
data using various techniques2 to identify patterns and correlations between the data
and which are used to establish a working model of relationships between inputs and
outputs. This model is gradually improved by iterative ‘learning’ that is, by testing its
predictions and correcting them when wrong, until it identifies something like what is
understood in conventional statistics as a ‘line of best fit’ to generate a model that
provides the strongest predictive relationship between inputs and outputs.3 Accordingly,
2 Five well-used techniques are logistic regression models, the naïve Bayes classifier, k-
nearest neighbors, decision trees and neural networks, all of which exemplify predictive
modelling: Mackenzie (2015) 432-433.
3 Machine learning algorithms can be broadly split into three categories based on how they
learn. Supervised learning requires a training data set with labeled data, or data with a known
4
this methodological approach is sometimes described as ‘letting the data speak’4 (Mayer-
Schonberger and Cukier 2013: 6). While conventional statistical regression models
worked with 10 or so different variables (such as gender, age, income, occupation,
educational level, income and so forth) and perhaps sample sizes of thousands, machine
learning algorithms that drive the kind of predictive analytic tools that are now
commonly in use are designed to work with hundreds (and sometimes tens of
thousands) of variables (‘features’) and sample sizes of millions or billions (Mackenzie
2015: 434).
Algorithmic decision-making refers to the use of algorithmically generated knowledge
systems to execute or inform decisions, and which can vary widely in simplicity and
sophistication. Algorithmic regulation refers to regulatory governance systems that
utilise algorithmic decision making. Although the scope and meaning of the term
‘regulation’ and ‘regulatory governance’ is contested (Baldwin et al 2010), I adopt the
definition offered by leading regulatory governance scholar Julia Black, who defines
regulation (or regulatory governance) as intentional attempts to manage risk or alter
behaviour in order to achieve some pre-specified goal (Black 2014). Several features of
this understanding of regulation are worth highlighting. First, although regulation is
output value. Unsupervised learning techniques do not use a training set, but find patterns or
structure in the data by themselves. Semi-supervised learning uses mainly unlabelled and a small
amount of labelled input data. Using a small amount of labelled data can greatly increase the
efficiency of unsupervised learning tasks. The model must learn the structure to organize the data
as well as make predictions (NESTA 2015: 5).
4 There is a growing literature in ‘Critical Data Studies’ (or ‘Critical Algorithm Studies’) which
seeks to explore data as situated in complex ‘data assemblages’ of action (Kitchen 2014b: 24-26),
referring to the vast systems, comprised not just of database infrastructures, but also the
‘technological, political, social and economic apparatuses that frames their nature, operation and
work’, including processes of data collection and categorization to its subsequent cleaning,
storing, processing, dissemination and application (Kitchen et al 2015). A growing body of
research examines the labour and political economies entailed in the reproduction of these
assemblages, using a wide range of disciplinary lenses including STS (Ziewieke 2016; Beer 2017;
MacKenzie 2015; Cheney-Lippold 2011) focusing on data from a variety of sources including
meteorological data, data produced by for-profit education companies, financial trading data and
biomedical data. This literature exposes the fallacy of understanding data as an objective set of
facts that exist prior to ideology, politics or interpretation by seeking to understand data as
situated in socio-technical systems that surround its production, processing, storing, sharing,
analysis and reuse, thereby demonstrating that the production of data assemblages is not a
neutral, technical process, but a normative, political and ethical one that is contingent and often
contested, with consequences for subsequent analysis, interpretation and action (Kitchen 2014a).
An illustration of the wide range of disciplinary perspectives, questions and approaches emerging
within this field can be found in Big Data & Society (2016), Special Issue on Critical Data Studies
available at http://bds.sagepub.com/content/critical-data-studies (accessed 11 November 2016).
5
widely regarded as a critical task of governments, regulation is also pursued by non-state
actors and entities (Black 2008). Just as a public transport authority may regulate vehicle
movement to optimise traffic flow, likewise a social media company such as Facebook
might regulate the posting and viewing behaviour of users to optimise its financial
returns. Secondly, the size of the regulated ‘population’ is highly variable. It may refer to
the intentional actions of one person who adopts some system or strategy to regulate
some aspect of her own behaviour (such as an individual who uses a fitness tracking
device to help her ensure that she attains a minimum level of daily physical activity)
through to regulatory systems that seek to direct and influence the behaviour of a large
number of people or entities, such as algorithmic systems employed by digital car-sharing
platforms, Uber, to enable drivers to offer motor vehicle transport services to individuals
at a pre-specified fee without having had any previous relationship. Thirdly, because
regulation is above all an intentional activity directed at achieving a pre-specified goal,
any regulatory system must have some kind of system ‘director’ (or ‘regulator’) to
determine the overarching goal of the regulatory system. Accordingly, I refer to
algorithmic regulation as decision-making systems that regulate a domain of activity in
order to manage risk or alter behaviour through continual computational generation of
knowledge from data emitted and directly collected (in real time on a continuous basis)
from numerous dynamic components pertaining to the regulated environment in order
to identify and, if necessary, automatically refine (or prompt refinement of) the system’s
operations to attain a pre-specified goal.
2.2 Forms of Algorithmic Regulation: A Taxonomy
Algorithmic regulation has antecedents in the interdisciplinary science of cybernetics that
emerged in the aftermath of World War II. Cybernetic analysis sought to move away
from linear understanding of cause and effect and towards investigation of control
through circular causality, or feedback (Medina 2015). The logic underpinning
algorithmic regulation, and the ‘smartification’ of everyday life which it makes possible,
rests on the continuous collection and analysis of primary data combined with metadata,
which logs the frequency, time and duration of device usage and which, via direct
machine-to-machine communication via digital networks, allows the combined data to be
algorithmically mined in order to trigger an automated response (Morozov 2014).
6
By understanding regulation as a cybernetic process involving the three core components
of any control system – i.e. ways of setting standards, goals or targets (‘standard-
setting’); ways of gathering information (‘information-gathering’) and ways of enforcing
those standards once deviation is identified in order to change behaviour so that it meets
the requisite standards (‘enforcement and behaviour modification’), various forms of
algorithmic regulation can be identified (Hood et al 2001: 23; Morgan and Yeung 2007:3 ).
My taxonomy identifies two alternative configurations for each component, thereby
generating a total of eight different forms (see Table 1). First, at the level of standard
setting, the behavioural norm which the system enforces may be either a simple, fixed
(yet reprogrammable) standard of behaviour. This is the most basic form of algorithmic
intervention, exemplified in the use of password protection systems to authorise access
to digital content. Alternatively, the behavioural standard may be adaptive, to facilitate
the attainment of whatever fixed, overarching (yet reprogrammable) system goal the
regulatory system is designed to optimise in order to produce system stability. These
latter systems are often described as ‘complex’ or ‘intelligent’, such as intelligent
transportation systems that effectively teach themselves how to identify the most
reliable predictor of traffic flow through machine learning processes that utilise trial and
error applied to continuously updated real-time traffic data (Lv et al 2015). Although
these systems allow behavioural variation, for example, in vehicle speed limits and/or the
duration, timing and frequency of traffic light cycles, depending upon traffic volume and
distribution, the overarching objective is pre-specified and fixed by the system director:
put simply, to optimise traffic flow within the system.5
Second, at the level of information gathering and monitoring, the system may operate on
a reactive basis, configured automatically to mine historic performance data in real-time
to detect violation. Simple reactive systems include automated vehicle speeding
detection systems that utilise speed cameras to provide real-time identification of
vehicles that exceed prescribed speed limits, while complex reactive systems include
credit card fraud detection systems that utilise machine learning techniques to profile the
5 Algorithmic systems of this kind also underpin growing labour market practices including
the use of ‘zero hour contracts’ which subject workers to variable scheduling, focusing paid work
hours to times of high demand thus shifting the risk of changing demand onto workers and
increasing work intensity (Wood 2016). Similarly, algorithmic workplace performance
management techniques rely on microlevel surveillance of call centre workers to provide feedback
to employers and employees aimed at optimising worker productivity (Kuchler 2014; Edwards and
Edwards 2016).
7
spending patterns of credit card holders, aimed at detecting suspicious transactions
when they occur, immediately alerting the credit provider and/or card-holder to take
action. Alternatively, algorithmic systems may be configured to detect violations on a
pre-emptive basis, applying machine learning algorithms to historic data to infer and
thereby predict future behaviour. Simple predictive systems include digital text auto-
complete systems while complex, pre-emptive systems make possible new forms of
personalised pricing, applying machine learning algorithms to data collected from on-line
tracking and measurement of on-line user behaviour at a highly granular level to generate
consumer profiles, varying the price of goods offered to individuals on-line, based on
algorithmic evaluations of the user’s willingness and ability to pay (Miller 2014).
Third, at the level of enforcement and behaviour modification, the system may
automatically administer a specified sanction or decision without any need for human
intervention beyond user input of relevant data (or data tokens), such as simple reactive
systems that automatically block access to web-content if the user fails to enter an
authorised password. These systems constitute a form of action-forcing (or coercive)
design (Yeung and Dixon-Woods 2010) thus offering the promise of immediate ‘perfect
enforcement’ (Zittrain 2009)6. These systems may also operate pre-emptively, based on
algorithmically determined predictions of a candidate’s future behaviour, such as systems
that automatically evaluate applications from individuals seeking access to services such
as loan finance, insurance cover and employment opportunities (O’Neil 2016). Although a
human operator might be authorised to review and override the automated decision at a
later stage, automation is relied upon to make and implement a decision that has real,
consequential effects for the individual.7 Alternatively, both simple and complex systems
6 Although the self-executing capacity of these systems holds considerable allure by
rendering human enforcement agents redundant, the legitimacy of ‘perfect enforcement’ has
been questioned by cyberlawyer Jonathan Zittrain who highlights the dangers of smart devices
(which he termed ‘tethered appliances’) emerging in an earlier internet age because they ‘invite
regulatory intervention that disrupts a wise equilibrium that depends upon regulators acting with
a light touch, as they traditionally have done within liberal societies’ (Zittrain 2009: 103).
Moreover, the promise of ‘perfect’ enforcement is illusory, given the inevitable impossibility of
defining ‘perfect’ standards that are capable of anticipating every single possible future event that
may be of relevance to the operation of the regulatory system’s goals (Yeung 2008: 92-93).
7 It is the use of these kinds of algorithmic decision-making systems that has given rise to
increasing concerns about errors in the underlying data, their application to particular individuals,
and their potential discriminatory effects. Concerns of this kind have spawned a rising chorus of
concern about the need for mechanisms that can secure algorithmic accountability, which are
discussed more fully at section 5 below.
8
may be configured to provide automated ‘assistance’ or ‘recommendations’ to a human
decision-maker, by prioritising candidates from within the larger regulated population.
These ‘recommender systems’ are intended to direct or guide an individual’s decision-
making processes in ways identified by the underlying software algorithm as optimal,
offering prompts that focus a human user’s attention on a particular set of entities within
the data set, with the human user retaining formal decision-making authority,
exemplified by on-line shopping recommendation engines (Yeung 2017).
Each of these forms of algorithmic regulation can be employed by state and non-state
institutions. Some systems are applied to regulate the conduct of many millions of
individuals, such as Facebook’s News Feed system (Luckerson 2015), while others may be
limited to managing relationships within a small group. In particular, when used to
manage the relationship between parties to a contract, they have been referred to by
others as computer mediated contracts, referring to arrangements between contracting
parties which harness the capacity of networked communication systems to undertake
continuous, real-time digital monitoring of behaviour to detect, monitor and enforce
performance of the terms of the contract, thereby overcoming a significant limitation of
conventional contracts : the need for the principal to monitor the behaviour of the agent
to guard against the agent’s temptation to ‘shirk’, that is, to act in a self-interested
manner that is contrary to the interests of the principal (Williamson 1975). Hal Varian,
Google’s Chief Economist, provides two examples of computer-mediated contracts that
significantly reduce these costs for the principal: first, remote vehicle monitoring systems
that verify whether driver behaviour conforms with the desired standard, thereby
enabling car rental companies to continuously monitor and verify whether a driver is
honouring his/her contractual obligation to operate the car in a safe manner. Secondly,
these vehicle monitoring systems enable automated remote enforcement that allow a
lender easily to repossess a car purchased by an individual on loan finance who fails to
make two consecutive monthly repayments by automatically immobilising the car (Varian
2014).
Table 1: A taxonomy of algorithmic regulatory systems
Standard Monitoring Enforcement/ Description
setting Sanction
1. Fixed Real time reactive Automated Simple real-time sanction
violation detection administration systems
2. Fixed Real time reactive Recommender Simple real-time warning
violation detection system systems
3. Fixed Pre-emptive Automated Simple pre-emptive sanction
9
