Defining and detecting quantum speedup

1.4 MB·

by  Troels F. Rønnow

Defining and detecting quantum speedup
Troels F. Rønnow,1 Zhihui Wang,2 Joshua Job,3 Sergio Boixo,4 Sergei V. Isakov,5
David Wecker,6 John M. Martinis,7 Daniel A. Lidar,8 and Matthias Troyer∗1
1Theoretische Physik, ETH Zurich, 8093 Zurich, Switzerland
2Department of Chemistry and Center for Quantum Information Science & Technology,
University of Southern California, Los Angeles, California 90089, USA
3Department of Physics and Center for Quantum Information Science & Technology,
University of Southern California, Los Angeles, California 90089, USA
4Google, 150 Main St, Venice Beach, CA, 90291
5Google, Brandschenkestrasse 110, 8002 Zurich, Switzerland
6Quantum Architectures and Computation Group, Microsoft Research, Redmond, WA 98052, USA
7Department of Physics, University of California, Santa Barbara, CA 93106-9530, USA
8Departments of Electrical Engineering, Chemistry and Physics,
and Center for Quantum Information Science & Technology,
4
1 University of Southern California, Los Angeles, California 90089, USA
0
2
The development of small-scale digital and analog quantum devices raises the question of how
n
a to fairly assess and compare the computational power of classical and quantum devices, and of
J how to detect quantum speedup. Here we show how to define and measure quantum speedup in

3 various scenarios, and how to avoid pitfalls that might mask or fake quantum speedup. We illustrate
1 our discussion with data from a randomized benchmark test on a D-Wave Two device with up to
503 qubits. Comparing the performance of the device on random spin glass instances with limited
]
precision to simulated classical and quantum annealers, we find no evidence of quantum speedup
h
p when the entire data set is considered, and obtain inconclusive results when comparing subsets of
- instances on an instance-by-instance basis. Our results for one particular benchmark do not rule out
t
n the possibility of speedup for other classes of problems and illustrate that quantum speedup is elusive
a and can depend on the question posed.
u
q
[
I. INTRODUCTION tumspeedupispolynomial,defininganddetectingquan-

1 tum speedup becomes more subtle. One such subtlety is
v how to properly define the hardness of a problem given
0 The interest in quantum computing originates in the
prior knowledge about the answer [6].
1 potential of a quantum computer to solve certain com-
9
putational problems much faster than is possible classi-
2
cally. Examples are the factoring of integers [1] or the
.
1 simulation of quantum systems [2]. Shor’s algorithm can
Here we discuss how to define “quantum speedup”
0 find the prime factors of an integer in a time that scales
4 and show that this term may refer to different quanti-
polynomially in the number of digits of the integer to be
1 ties depending on the goal of the study. In particular,
factored, while all known classical algorithms scale ex-
: we define what we call “limited quantum speedup”—
v ponentially. The simulation of the time evolution of a
i essentially a speedup relative to a given, corresponding
X quantum system on a classical computer also takes ex-
classicalalgorithm—andexplainhowsuchaspeedupcan
ponential resources, because the Hilbert space of an N
r be reliably detected. To illustrate these issues we com-
a particle system is exponentially large in N, while quan-
paretheperformanceofa503-qubitD-WaveTwo(DW2)
tumhardwarecansimulatethesametimeevolutionwith
devicetoclassicalalgorithmsrunningonastandardCPU
polynomial complexity [3, 4].
and analyze the evidence for quantum speedup on ran-
In these examples the quantum algorithm is exponen- dom spin glass problems. This example is particularly
tially faster than the best available classical algorithm. relevant since it is an open question whether quantum
This type of exponential quantum speedup substantially annealing [7] or the quantum adiabatic algorithm [8] can
simplifiesthediscussion,sinceitrendersthedetailsofthe exhibit a quantum speedup for such problems. Random
classical or quantum hardware unimportant. According spin glass problems are an interesting benchmark prob-
to the extended Church-Turing thesis all classical com- lem, though not necessarily the most relevant for practi-
puters are equivalent up to polynomial factors [5]. Sim- cal applications, such as machine learning. We also dis-
ilarly, all proposed models of quantum computation are cuss issues that might mask or fake a quantum speedup
polynomially equivalent, so that a finding of exponential whennotconsideredcarefully, suchascomparingsubop-
quantum speedup will be model-independent. In other timalalgorithmsorimproperlyaccountingforthescaling
cases, in particular on small devices, or when the quan- of hardware resources.
2
II. DEFINING QUANTUM SPEEDUP consensus may be time- and community-dependent [14].
Intheabsenceofaconsensusaboutwhatisthebestclas-
A. The classical to quantum scaling ratio sicalalgorithm,wedefinepotential (quantum) speedupas
a speedup compared to a specific classical algorithm or
a set of classical algorithms. An example is the simula-
When the time to solution depends not only on the
tionofthetimeevolutionofaquantumsystem,wherethe
problemsizeN butalsoonthespecificprobleminstance,
propagationofthewavefunctiononaquantumcomputer
thenthepurposeofthecomparisonbecomesanotherfac-
wouldbeexponentiallyfasterthanadirectintegrationof
torindecidinghowtomeasureperformance. Specifically,
Schr¨odinger’sequationonaclassicalcomputer. Apoten-
whenadeviceisusedasatoolforsolvingproblems,then
tial quantum speedup can of course be trivially attained
the question of interest is to determine which device is
by deliberately choosing a poor classical algorithm (for
better for the hardest problem, or for almost all possible
example,factoringusingclassicalinsteadofquantumpe-
problem instances. On the other hand, if we are inter-
riodfindingwhileignoringknown,betterclassicalfactor-
ested in aspects of the underlying physics of a device
ingalgorithms),sothatheretooonemustmakeagenuine
then it might suffice to find some instances or a subclass
attempttocompareagainstthebestclassicalalgorithms
of instances where a quantum device exhibits a speedup.
known, and any potential quantum speedup might be
These two questions will lead to different quantities of
short-lived if a better classical algorithm is found.
interest.
Underlying all the above notions of quantum speedup
Inallofthesecaseswedenotethetimeusedbyaclas-
is the availability of a fully coherent, universal quantum
sical device to solve a problem of size N by C(N) and
computer. A weaker scenario is one where the device
the time used on the quantum device by Q(N), defining
is merely a putative or candidate quantum information
quantum speedup as the ratio
processor, or where a quantum algorithm is designed to
C(N) makeuseofquantumeffectsbutitisnotknownwhether
S(N)= . (1)
Q(N) thesequantumeffectsprovideanadvantageoverclassical
algorithms. To capture this scenario, which is of central
Note that in both the quantum and classical case this
interest to us in this work, we define limited quantum
definition includes a specific choice of algorithm and de-
speedup as a speedup obtained when comparing specifi-
vice.
cally with classical algorithms that “correspond” to the
The first question that arises is which classical algo-
quantumalgorithminthesensethattheyimplementthe
rithm to compare against, i.e., what is C(N). This leads
same algorithmic approach, but on classical hardware.
to different definitions of quantum speedup.
In the context of an analog quantum device this can be
thought of as being the result of decohering the device.
Since there is no unique way to decohere a quantum de-
B. Five different types of quantum speedup
vice, one may arrive at different corresponding classical
algorithms. Anaturalexampleisquantumannealingim-
The optimal scenario is one of a provable quantum
plemented on a candidate physical quantum information
speedup,wherethereexistsaproofthatnoclassicalalgo-
processor vs either classical simulated annealing, classi-
rithm can outperform a given quantum algorithm. Per-
cal spin dynamics, or simulated quantum annealing (as
haps the best known example is Grover’s search algo-
defined in Methods). In this comparison a limited quan-
rithm [9], which exhibits a provable quadratic speedup
tum speedup would be a demonstration that quantum
over the best possible classical algorithm [10], assuming
effects improve the annealing algorithm [15].
an oracle.
Astrong quantum speedup wasdefinedin[11]byusing
theperformanceofthebest classicalalgorithmforC(N),
III. CLASSICAL AND QUANTUM ANNEALING
whether such an algorithm is known or not. Unfortu-
OF A SPIN GLASS
nately the performance of the best classical algorithm
is unknown for many interesting problems. In the case
As our primary example we will use the problem of
of factoring, for example, all known classical algorithms
finding the ground state of an Ising spin glass model de-
have super-polynomial cost in the number of digits N of
scribed by a “problem Hamiltonian”
the number to be factored [12], while the cost of Shor’s
algorithm is polynomial in N. However, a proof of a H =−(cid:88)h σz− (cid:88) J σzσz , (2)
Ising i i ij i j
classical exponential lower-bound for factorization is not
i∈V (i,j)∈E
known [13]. A less ambitious goal is therefore desirable,
and thus one usually defines quantum speedup (without with N binary variables σz = ±1. The local fields {h }
i i
additional adjectives) by comparing to the best available andcoupling{J }arefixedanddefineaprobleminstance
ij
classical algorithm instead of the best possible classical of the Ising model. The spins occupy the vertices V of a
algorithm. graph G={V,E} with edge set E. Solving this problem
However, this notion of quantum speedup depends on problemisNP-hardalreadyforplanargraphs[16],which
there being a consensus about “best available”, and this means that no polynomial time algorithm to find these
3
ground states is known and the computational effort of
all existing classical algorithms scales exponentially with
problem size. NP-hardness refers only to the hardest
106
problems, but the typical problem in our benchmarks, A) SA, median
where the graph forms a two-dimensional (2D) lattice, 105
is still hard since for zero local fields (hi = 0) there ex- 104
icsrtisticaalsptienmgplearsastuprheasTecat=ze0rofotremthpeesreat2uDre.spWinhgilleastshees e []µs 103
makes the problem easier than 3D spin glasses with a m 102
nisonnezveerortThecl>ess0n[o1n7]-,trsiovlivailnagntdhewtityhpiaclallkpnroowbnlemalgionrsittahnmces otal ti 101
T Annealing time [MCS]
a super-polynomial scaling is observed. While quantum 100
5 100 1000
mechanics is not expected to reduce this scaling to poly- 10-1 10 200 2000
nomial,aquantumalgorithmmightstillscalebetterwith 50 500 True scaling
10-2
problem size N than any classical algorithm.
8 32 72 128 200 288 392 512
Weusesimulatedannealing(SA)[18],simulatedquan- p p p p p p p p
tum annealing (SQA) [19, 20], and a DW2 device to find 108
B) SQA, median
the ground states of the Ising model above (see Methods
107
for details). The D-Wave devices [21–24] are designed
to be physical realizations of quantum annealing using 106
superconductingfluxqubitsandprogrammablecouplers. ]µs 105
Testsona108-qubitD-WaveOne(DW1)device[25]have e [
m 104
smhaolwbnatthh,atthdeesdpeivtiecedeccoorhreelraetnecsewaenldl wciotuhpSliQngAt,owahitchheris- otal ti 103
consistent with it actually performing quantum anneal- T 102 Annealing time [MCS]
ing[26,27]. Italsocorrelateswellwiththepredictionsof 5 100 1000
aquantummasterequation[28],whichisconsistentwith 101 1500 250000 2T0ru0e0 scaling
it being governed by open system quantum dynamics. It 100
iswellunderstoodthattheD-Wavedevices,justlikeany 8 32 72 128 200 288 392 512
other quantum information processing device, must be p p pLinearp problepm size pN p p
error-corrected in order to overcome the effects of deco- p
herence and control errors. While such error correction
FIG. 1. Scaling of the typical time to find a solution
hasalreadybeendemonstrated[29],ourstudyfocuseson
at constant annealing time. Shown is the typical (me-
the native performance of the device.
dian) time to find a ground state with 99% probability for
All annealing methods mentioned above are heuristic. spin glasses with ±1 couplings and no local field. A) for SA,
They are not guaranteed to find the global optimum in B)forSQA.Theenvelopeofthecurvesatconstantta,shown
a single annealing run, but only find it with a certain in red, corresponds to the minimal time at a given problem
instance-dependent success probability s≤1. We deter- size N and is relevant for discussion of the asymptotic scal-
ing. Annealing times are given in units of Monte Carlo steps
mine the true ground state energy using an exact belief
(MCS). One MCS corresponds to one update per spin. Note
propagation algorithm [30]. We then perform at least
in particular that the slope for small N is much flatter at
1000repetitionsoftheannealingforeachinstance,count
large annealing time (e.g., MCS = 4000) than that of the
howoftenthegroundstatehasbeenfoundbycomparing
true scaling.
to the exact result, and use this to estimate the success
probability s for each problem instance.
The total annealing time is defined as the time to per-
form R annealing runs, where R is the number of repeti-
tions needed to find the ground state at least once with
probability p: IV. CONSIDERATIONS WHEN COMPUTING
QUANTUM SPEEDUP
(cid:24) (cid:25)
log(1−p)
R= (3)
log(1−s)
Let us first consider the subtleties of estimating the
asymptoticscalingfromsmallproblemsizesN,andinef-
In order to reduce the effect of calibration errors on the ficiencies at small problem sizes that can fake or mask a
DW2, it is advantageous to repeat the annealing runs speedup. In the context of annealing methods the opti-
for several different encodings (“gauges”) of a problem mal choice of the annealing time turns out to be crucial
instance. See Methods for details. for estimating asymptotic scaling.
4
6 timal performance at small problem sizes N, and should
Suboptimal thereforenotbeinterpretedasspeedup. Toillustratethis
5 Optimal we show in Figure 2 (solid line) the true “speedup” ratio
ofthescalingofSAandSQA(actuallyaslowdown),and
4 a misleading, fake speedup (dashed line) due to using a
30 constantandexcessivelylongannealingtimet forSQA.
1 a
·
A Since the initial, slow increase of the total SQA effort at
Q 3
S
T constant annealing time is a lower bound for the scaling
/
A
S of the true effort, the speedup slope obtained from this
T 2
data—which depends inversely on the SQA effort—is an
upper bound, as confirmed by Figure 2.
1
0
8 32 72 128 200 288 392 512 B. Resource usage and speedup from parallelism
p p pLinearp problepm size pN p p
p A related issue is the scaling of hardware resources
FIG. 2. Pitfalls when detecting speedup. Shown is the with problem size and parallelism in classical and quan-
speedupofSQAoverSA,definedastheratioofmediantime tum devices. To avoid mistaking a parallel speedup for
tofindasolutionwith99%probabilitybetweenSAandSQA. a quantum speedup we need to scale hardware resources
Twocasesarepresented: a)bothSAandSQArunoptimally
(computational gates and memory) in the same way for
(i.e., the ratio of the true scaling curves shown in Figure 1),
the devices we compare, and employ these resources op-
andthereisnoasymptoticspeedup(solidline). b)SQAisrun
timally. These considerations are not universal but need
suboptimallyatsmallsizesbychoosingafixedlargeannealing
tobecarefullyappliedforeachcomparisonofaquantum
time t = 10000 MCS (dashed line). The apparent speedup
a
algorithm and device to a classical one.
is, however, due to suboptimal performance on small sizes
and not indicative of the true asymptotic behavior given by For a problem of size N, the DW2 uses only N out of
the solid line, which displays a slowdown of SQA compared 512 qubits and O(N) couplers and classical logical con-
to SA. trolgatestosolveaspinglassinstancewithN spinvari-
ables. Wedenotethetimeitneedstosolveaproblemby
T (N). The classical simulated annealer (or simulated
DW
A. Asymptotic scaling: SA vs SQA
quantum annealer) running on a single classical CPU,
on the other hand, uses fixed resources independent of
To illustrate these issues we consider the time to so- problem size N, and we denote the time it requires to
lution using SA and SQA run at different fixed anneal- solve a problem by T (N). We consider here only the
SA
ing times ta, independent of the problem size N. The pure annealing times, as they are what is relevant for
problem instances we choose are random couplings of the asymptotic scaling rather than the readout or setup
Jij =±1oneachoftheedgesinaperfectChimeragraph times, which scale subdominantly for large problems.
of L×L unit cells, containing N =8L2 spins (see Meth- Inordertoavoidconfusingquantumspeedupwithpar-
ods). We set the local fields hi = 0. Figure 1 shows the allel speedup we thus consider as a classical counterpart
scalingofthemediantotalannealingtime(over1000dif- totheDW2a(hypothetical)specialpurposeparallelclas-
ferent random instances) for both SA and SQA to find sicalsimulatedannealingdevice,withthesamehardware
a solution with probability p = 0.99. We observe that scalingastheDW2. Simulatedannealing(andsimulated
at constant ta, as long as ta is long enough to find the quantum annealing) is perfectly parallelizable for the bi-
ground state almost every time, the scaling of the total partite Chimera graphs realized by the DW2. The rea-
effort is at first relatively flat. The total effort then rises son is that one Monte Carlo step (consisting of one at-
more rapidly, once one reaches problem sizes for which tempted update per spin) can be performed in constant
the chosen annealing time is too short and the success time, since all spins in each of the two sublattices can
probabilities are thus low, requiring many repetitions. be updated simultaneously. The time to solve a problem
Figure1demonstratesthatnoconclusioncanbedrawn on this equivalent classical device, denoted by T (N), is
C
from annealing (simulated or in a device) about the thus related to the time T (N) taken by a simulated
SA
asymptotic scaling at fixed annealing times. It is mis- annealer using a fixed-size classical CPU by
leading to conclude about the asymptotic scaling from
the initial slow increase at constant t , and instead the 1
a T (N)∝ T (N), (4)
optimal annealing time toapt needs to be found for each C N SA
problemsizeN [25,31]. Thelowerenvelopeofthescaling
curves (indicated in red in Figure 1) corresponds to the since the latter needs time O(N) for one Monte Carlo
total effort at an optimal size-dependent annealing time step, while the former performs it in constant time.
topt(N) and can be used to infer the asymptotic scaling. The quantum part of speedup is then estimated by
a
In fact, the initial, relatively flat slope is due to subop- comparing the times required by two devices with the
5
same hardware scaling, giving of the couplings J from 2r discrete values {n/r}, with
ij
n ∈ {−r,−r−1,...,−1,1,...,r−1,r}, and call r the
S(N)= TC(N) ∝ TSA(N) 1 . (5) “range”. Thus when the range r =1 we only pick values
T (N) T (N)N J =±1. This choice is the least susceptible to calibra-
DW DW ij
tion errors of the device, but the large degeneracy of the
Thefactor1/N inthespeedupcalculationthusdiscounts groundstatesinthesecasesmakesfindingagroundstate
for the intrinsic parallel speedup of the analog device somewhateasier. Attheoppositeendweconsiderr =7,
whose hardware resources scale as N. See Methods for whichistheupperlimitgiventhefourbitsofaccuracyof
an alternative derivation that leads to the same results the couplings in the DW2. These problem instancess are
(uptosubleadingcorrections)byusingafixedsizedevice hardersincetherearefewerdegenerateminima,butthey
efficiently. also suffer more from calibration errors in the device. In
theSupplementaryMaterialwepresentadditionalresults
for r =3.
V. PERFORMANCE OF D-WAVE TWO
VERSUS SA AND SQA
C. Performance as an optimizer: comparing the
A. Comparing devices scaling of hard problem instances
If the goal is to compare the performance of devices 1. Pure annealing time
as optimizers, then one is interested in solving almost
all problem instances. In this case we should run the We start our analysis by focusing on pure annealing
devices in such a way that all but a small fraction of timesandshowinFigure3thescalingofthetimetofind
the problems can be solved. This will lead to a speedup the ground state at least once with probability p = 0.99
defined as the ratio of the quantiles (“R of Q”) of the for various quantiles, from the easiest instances (1%) to
timetosolution,withanemphasisonthehighquantiles, the hardest (99%), for two different ranges. Since we
whichwediscussinSec.VC.Acomplementaryquestion do not a priori know the hardness of a given problem
is to ask whether a device exhibits better performance instance we have to assume the worst case and perform
thananotherforsome problems. Toanswerthisquestion a sufficient number of repetitions R to be able to solve
we compare the time to solution individually for each even the hardest problem instances. Hence the scaling
problem instance. We then consider the quantiles of the for the selected high quantile will apply to all problem
ratio (“Q of R”) of the time to solution, and discuss this instances we run on the optimizer.
approach in Sec. VD. Inallthreecases(SA,SQA,DW2)weobserve,forsuffi-
Acomplementarydistinctionisthatbetweenwall-clock cientl√ylargeN,thatthetotaltimetosolutionscaleswith
time,denotingthefulltimetosolution,andthepure an- exp(c N), as reported previously for SA and SQA [25].
√
nealing time. Wall-clock time is the total time to find a The origin of the N exponent is well understood for
solution and is the relevant quantity when one is inter- exact solvers as reflecting the treewidth of the Chimera
ested in the performance of a device for applications and graph (see Methods and Ref. [34]), and a similar scaling
hasbeenusedinRef. [32]. Itincludesthesetup,cooling, is observed here for the heuristic algorithms. While the
annealingandreadouttimesontheDW2,andthesetup, SA and SQA codes were run at an optimized annealing
annealingandmeasurementtimefortheclassicalanneal- time for each problem size N, the DW2 has a minimal
ing codes. Thepure annealing time is simply Rta, where annealing time of ta = 20µs, which is longer than the
R is the number of repetitions and ta the time used for optimaltimeforallproblemsizes(seeMethods). There-
a single annealing run. It is the relevant quantity when fore the observed slope of the DW2 data should only be
one is interested in the intrinsic physics of the annealing taken as a lower bound for the asymptotic scaling. Even
processesandinscalingtolargerproblemsizesonfuture so, we observe similar scaling for the classical codes and
devices. We discuss both wall-clock and pure annealing on DW2.
times below.
2. The ratio of quantiles
B. Problem instances
With algorithms such as SA or quantum annealing,
The family of problem instances we use for our bench- where the time to solution depends on the problem in-
marking tests employ couplings J on all edges of N = stance, itisoftennotpossible(andusuallyirrelevant)to
ij
8LL(cid:48)-vertexsubgraphsoftheChimeragraphoftheDW2, experimentally find the hardest problem instance. It is
comprising L×L(cid:48) unit cells, with L,L(cid:48) ∈{1,...,8}. We preferable to decide instead for which fraction of prob-
set the fields h = 0 since nonzero values of the fields h lem instances one wishes to find the ground state, which
i i
destroythespinglassphasethatexistsatzerofield,thus then defines the relevant quantile. If we target q% of
making the instances easier [33]. We choose the values the instances then we should consider the qth percentile
6
108 108
107 A) SA, range 1 107 B) SA, range 7
106 106
105
]µs 104 ]µs 105
me [ 103 me [ 104
otal ti 110012 otal ti 110023
T 100 T
10-1 101
10-2 100
10-3 10-1
8 32 72 128 200 288 392 512 8 32 72 128 200 288 392 512
p p p p p p p p p p p p p p p p
1011 1011
C) SQA, range 1 D) SQA, range 7
1010 1010
109 109
]µs 108 ]µs 108
e [ 107 e [ 107
m m
al ti 106 al ti 106
ot 105 ot 105
T T
104 104
103 103
102 102
8 32 72 128 200 288 392 512 8 32 72 128 200 288 392 512
p p p p p p p p p p p p p p p p
108 109
E) DW, range 1 F) DW, range 7
107 108 99% 50%
106 107 95% 10%
90% 5%
]µs 105 ]µs106 75% 1%
e [ 104 e [ 105
m m
al ti 103 al ti 104
ot 102 ot 103
T T
101 102
100 101
10-1 100
8 32 72 128 200 288 392 512 8 32 72 128 200 288 392 512
p p pLinearp problepm size pN p p p p pLinearp problepm size pN p p
p p
FIG.3. Scalingoftimetosolutionfortherangesr=1(panelsA,CandE)andr=7(panelsB,DandF).Shown
is the scaling of the time to find the ground state at least once with a probability p = 0.99 for various quantiles of hardness,
for A,B) simulated annealing (SA), C,D) simulated quantum annealing (SQA) and E,F) the DW2. The SA and SQA data is
obtainedbyrunningthesimulationsatanoptimizedannealingtimeforeachproblemsize. TheDW2annealingtimeof20µsis
theshortestpossible. Notethedifferentverticalaxisscales,andthatboththeDW2andSQAhavetroublesolvingthehardest
instances for the large problem sizes, as indicated by the terminating lines for the highest quantiles. More than the maximum
number of of repetitions (10000 for SQA, at least 32000 for DW2) of the annealing we performed would be needed to find the
ground state in those cases.
in the scaling plots shown in Figure 3. The appropriate the interesting regime of large N. That is, while for all
speedupquantityisthentheratio of these quantiles. De- quantiles, and for both ranges (with the exception of the
noting a quantile q of a random variable X by [X] we 50th quantile and r = 1), the initial slope is positive,
q
can define this as when N becomes large enough we observe a turnaround
andeventuallyanegativeslope,showingthatSAoutper-
[T (N)] [T (N)] 1
SRofQ(N)= C q ∝ SA q . (6) forms the DW2.
q [T (N)] [T (N)] N
DW q DW q
Taking into account that (as discussed in Sec. IVA)
Plotting this quantity for the DW2 vs SA in Figure 4 duetothefixedsuboptimalannealingtimesthespeedup
we find no evidence for a limited quantum speedup in defined in Eq. (6) is an upper bound, we conclude that
7
1.0 108
A) Range 1 A) Range 1
107
0.8
N
2/ ]s 106
1 µ
]5Wq·0.6 me [ 105
/[TD0.4 al ti 104
]SAq Tot 103
[T 0.2 102
0.0 101
8 32 72 128 200 288 392 512 8 32 72 128 200 288 392 512
p p p p p p p p p p p p p p p p
5 108
B) Range 7 B) Range 7
50%
4 75% 107 99% 75%
]512/NWq·3 999059%%% me []µs 110056 9950%% 50%
]/[TSAqD 2 Total ti 104
[T 1 103
0 102
8 32 72 128 200 288 392 512 8 32 72 128 200 288 392 512
p p pLinearp problepm size pN p p p p pLinearp problepm size pN p p
p p
FIG. 4. Speedup for ratio of quantiles for the DW2 FIG.5. Comparing wall-clock timesAcomparisonofthe
compared to SA. A) For instances with range r = 1. B) wall-clock time to find the solution with probability p=0.99
For instances with range r = 7. Shown are curves from the for SA running on a single CPU (dashed lines) compared to
median (50th quantile) to the 99th quantile. 16 gauges were the DW2 (solid lines) using 16 gauges. A) for range r = 1,
used. In these plots we multiplied Eq. (6) by 512 so that B) for range r=7. Shown are curves from the median (50th
the speedup value at N = 512 directly compares one DW2 quantile) to the 99th quantile. The large constant program-
processor against one classical CPU. mingoverheadoftheDW2maskstheexponentialincreaseof
timetosolutionthatisobviousintheplotsofpureannealing
time. ResultsforasinglegaugeareshownintheSupplemen-
tary Material.
the DW2 does not exhibit a speedup over SA for this
particular benchmark.
D. Instance-by-instance comparison
1. Total time to solution
We now focus on the question of whether the DW2
exhibits a limited quantum speedup for some fraction of
3. Wall-clock time the instances of our benchmark set. To this end we per-
form individual comparisons for each instance and show
in Figure 6A-B the ratios of time to solution between
While not as interesting from a complexity theory the DW2 and SA, considering only the pure annealing
point of view, it is instructive to also compare wall-clock time. We find a wide scatter, which is not surprising
timesfortheabovebenchmarks,aswedoinFigure5. We sincewepreviouslyfoundthatDW1performslikeasim-
observe that the DW2 performs similarly to SA run on a ulated quantum annealer, but correlates less well with a
single classical CPU, for sufficiently large problem sizes simulated classical annealer [25]. We find that while the
and at high range values. Note that the large constant DW2 is sometimes up to 10× faster in pure annealing
programming overhead of the DW2 masks the exponen- time, there are many cases where it is ≥100× slower.
tial increase of time to solution that is obvious in the Considering the wall-clock times, the advantage of the
plots of pure annealing time. DW2 seen in Figure 6A-B for some instances tends to
8
Pure annealing time Wall-clock time Wall-clock time
averaging 16 gauges single gauge 16 gauges
105 105 105 20
A) Range 1 C) Range 1 E) Range 1
104 104 104
s] 18
m
[W 103 103 103
D
e T 102 102 102 16
m
al ti 101 101 101 14
ot
T 100 100 100
12
10-1 10-1 10-1 s
10-1 100 101 102 103 104 105 10-1 100 101 102 103 104 105 10-1 100 101 102 103 104 105 ce
10an
st
105 105 105 n
B) Range 7 D) Range 7 F) Range 7 I
8
104 104 104
s]
m
[W 103 103 103 6
D
me T 102 102 102 SA faster 4
al ti 101 101 101
ot 2
T 100 100 100 DW2 faster
10-1 10-1 10-1 0
10-1 100 101 102 103 104 105 10-1 100 101 102 103 104 105 10-1 100 101 102 103 104 105
Total time T [ms] Total time T [ms] Total time T [ms]
SA SA SA
FIG. 6. Instance-by-instance comparison of annealing times and wall-clock times. Shown is a scatter plot of the
pureannealingtimefortheDW2comparedtoasimulatedclassicalannealer(SA)usinganaverageover16gaugesontheDW2.
A) DW2 compared to SA for r = 1, B) DW2 compared to SA for r = 7. The color scale indicates the number of instances
in each square. Instances below the diagonal red line are faster on the DW2, those above are faster classically. Instances for
whichtheDW2didnotfindthesolutionwith10000repetitionspergaugeareshownatthetopoftheframe(nosuchinstances
were found for SA). Panels C) and D) show wall-clock times using a single gauge on the DW2. Panels E) and F) show the
wall-clock time for DW2 using 16 gauges. N =503 in all cases.
disappear, since it is penalized by the need for program- whichcangiveinsightintothebehavioroffuturedevices
mingthedevicewithmultipledifferentgaugechoices(see thatcansolvelargerproblems. InSectionVCwedidnot
Methods). Figure 6C-D shows that for one gauge choice find evidence for a limited quantum speedup when con-
there are some instances, for r = 7, where the DW2 is sidering all instances. Now we consider instead whether
faster,butmanyinstanceswhereitneverfindsasolution. thereissuchaspeedupforasubsetofprobleminstances.
Using16gaugestheDW2findsthesolutioninmostcases, Tothisendwestudythescalingoftheratiosofthetime
butisalwaysslowerthantheclassicalannealeronaclas- to solution for individual instances, and display in Fig-
sical CPU for r = 1, as can be seen in Figure 6E-F. For ure 7 the scaling of various quantiles of the ratio
r =7theDW2issometimesfasterthanasingleclassical (cid:20) (cid:21) (cid:20) (cid:21)
T (N) T (N) 1
CPU. Overall, the performance of the DW2 is better for SQofR(N)= C ∝ SA . (7)
q T (N) T (N)N
r = 7 than for r = 1, and comparable to SA only when DW q DW q
justthepureannealingtimeisconsidered. Thedifference
For r = 7 all the quantiles bend down for sufficiently
to the results of Ref. [32] is due to the use of optimized
large N, so that there is no evidence of a limited quan-
classical codes using a full CPU in our comparison, as
tum speedup. Yet, now there seems to be an indication
opposed to the use of generic optimization codes using
of such a speedup compared to SA in the high quan-
only a single CPU core in Ref. [32].
tiles for r = 1. However, for the reasons discussed in
Sec. IVA, one must be careful not to overinterpret this
as solid evidence for a speedup since the instances con-
2. Quantiles of ratio tributingherearenotrunattheoptimalannealingtime.
Moreover, as discussed in the Supplementary Material,
Comparisonsoftheabsolutetimetosolutionareoflim- we find no evidence of a limited quantum speedup for
itedimportancecomparedtotherealquestionofscaling, r = 3. Thus, while perhaps encouraging from the per-
9
portedbytheSAandSQAdatashowninFigure1,andis
plausible as long as the growing annealing time does not
103 becomecounterproductiveduetocouplingtothethermal
102 A) Range 1 bath[35]. Bydefinition,TDW(N,toapt(N))≤TDW(N,ta),
101 where we have added the explicit dependence on the an-
N
2/ 100 nealing time, and ta is a fixed annealing time. Thus
1
5·10-1
]DWq10-2 S(N)= TC(N) 1 (8)
/TA10-3 TDW(N,ta)N
S
[T 10-4 ≤ TC(N) 1 =Sopt(N).
10-5 T (N,topt(N))N
DW a
10-6
8 32 72 128 200 288 392 512 Underourassumption, topt(N)<t forsmall N, but for
p p p p p p p p a a
sufficiently large N the optimal annealing time grows so
102 B) Range 7 thattopt(N)≥t . ThustheremustbeaproblemsizeN∗
a a
101 at which toapt(N∗) = ta, and hence at this special prob-
N lem size we also have S(N∗) = Sopt(N∗). However, as
2/ 100
1 mentioned in Section VC1, the minimal annealing time
5
]Wq·10-1 of 20µs is longer than the optimal time for all problem
TD10-2 sizes(seeSupplementaryMaterial),i.e.,N∗ >503inour
/
[TSA10-3 99% 75% 5% scuasffie.ciTenhtelyrelfaorrgee,iNfS,(aNsw)eisianddeeecdreoabssinergvfeuinnctailolnouorfN“Rfoorf
95% 50% 1%
10-4 Q” results (recall Figure 4), then since Sopt(N)≥S(N)
90% 10%
and S(N∗)=Sopt(N∗),itfollowsthatSopt(N)toomust
8 32 72 128 200 288 392 512
be a decreasing function for a range of N values, at least
p p pLinearp problepm size pN p p
untilN∗. Thisshowsthattheslowdownconclusionholds
p
also for the case of optimal annealing times.
FIG. 7. Speedup for quantiles of the ratio of the DW2 For the instance-by-instance comparison (“Q of R”),
compared to SA, for A) r = 1, B) r = 7. No asymptotic no such conclusion can be drawn for the subset of in-
speedupisvisibleforanyofthequantilesatr=7,whilesome
stances (at r = 1) corresponding to the high quantiles
evidenceofalimitedquantumspeedup(relativetoSA)isseen
whereSQofR(N)isanincreasing functionofN. Thislim-
forquantileshigherthanthemedianatr=1. AsinFigure4 q
ited quantum speedup may or may not persist for larger
we multiplied Eq. (7) by 512 so that the speedup value at
problem sizes or if optimal annealing times are used.
N = 512 directly compares one DW2 processor against one
classical CPU.
VI. DISCUSSION
spective of a search for a (limited) quantum speedup,
more work is needed to establish that the r = 1 result In this work we have discussed challenges in prop-
persistsforthoseinstancesforwhichonecanbesurethat erly defining and assessing quantum speedup, and used
the annealing time is optimal. comparisonsbetweenaDW2andsimulatedclassicaland
quantum annealing to illustrate these challenges. Strong
or provable quantum speedup, implying speedup of a
E. Arguments for and against a speedup on the quantum algorithm or device over any classical algo-
DW2 rithm, is an elusive goal in most cases and one thus usu-
ally defines quantum speedup as a speedup compared to
Let us consider in some more detail the speedup re- the best available classical algorithm. We have intro-
sultsdiscussedabove. Wehavearguedthattheapparent duced the notion of limited quantum speedup, referring
limitedquantumspeedupseeninther =1resultsofFig- toamorerestrictedcomparisonto“corresponding”clas-
ure 7 must be treated with care due to the suboptimal sicalalgorithmssolvingthesametask,suchasaquantum
annealingtime. Itmightthenbetemptingtoarguethat, annealer compared to a classical annealing algorithm.
strictly speaking, the comparison with suboptimal-time Quantum speedup is most easily defined and detected
instancescannotbeusedforclaimingaslowdowneither, in the case of an exponential speedup, where the de-
i.e., that we simply cannot infer how the DW2 will be- tails of the quantum or classical hardware do not matter
haveforoptimal-timeinstancesbybasingtheanalysison since they only contribute subdominant polynomial fac-
suboptimal times only. tors. Inthecaseofanunknownorapolynomialquantum
However, let us make the assumption that, along with speedup one must be careful to fairly compare the clas-
the total time, the optimal annealing time topt(N) also sical and quantum devices, and, in particular, to scale
a
grows with problem size N. This assumption is sup- hardware resources in the same manner. Otherwise par-
10
allel speedup might be mistaken for (or hide) quantum sical algorithms for the problem class we have studied
speedup. [17]; or, perhaps, the noisy implementation in the DW2
An experimental determination of quantum speedup cannot realize quantum speedup and is thus not better
suffers from the problem that all measurements are lim- thanclassicaldevices. Alternatively, aspeedupmightbe
ited to finite problem sizes N, while we are most inter- masked by calibration errors, improvements might arise
ested in the asymptotic behavior for large N. To arrive fromerrorcorrection[29],orotherproblemclassesmight
atareliableextrapolationitisadvantageoustofocusthe exhibit a speedup [39]. Future studies will probe these
scalinganalysisonthepartoftheexecutiontimethatbe- alternatives and aim to determine whether one can find
comes dominant for large problem sizes N, which in our a class of problem instances for which an unambiguous
example is the pure annealing time, and not the total speedup over classical hardware can be observed.
wall-clock time. For each problem size we furthermore
need to ensure that neither the quantum device nor the
classicalalgorithmarerunsuboptimally,sincethismight METHODS
hide or fake quantum speedup.
If the time to solution depends not only on the prob- Simulated annealing. Simulated annealing [18] per-
lem size N but also on the specific problem instance, forms a Monte Carlo simulation on the model of Eq. (2),
then one needs to carefully choose the relevant quantity starting from a random initial state at high temperature.
During the course of the simulation the temperature is
to benchmark. We argued that in order to judge the
lowered towards zero. At the end of the annealing schedule,
performance over many possible inputs of a randomized
at low temperature, the spin configuration of the system
benchmark test, one needs to study the high quantiles,
ends up in in a local minimum. By repeating the simulation
and define speedup by considering the ratio of the quan-
manytimesonemayhopetofindtheglobalminimum. More
tiles of time to solution. If, on the other hand, one is specifically,SAisperformedbysequentiallyiteratingthrough
interested in finding out whether there is a speedup for all spins and proposing to flip them based on a Metropolis
some subset of problem instances, then one can instead algorithm using the Boltzmann weight of the configuration
perform an instance-by-instance comparison by focusing at finite temperature. During the annealing schedule we
on the quantiles of the ratio of time to solution. linearly increase the inverse temperature over time from an
initial value of β =0.1 to a final value of β =3r.
We note that it is not yet known whether a quantum
For the case of ±1 couplings (r = 1), and for r = 3 we
annealer or even a perfectly coherent adiabatic quantum
use a highly optimized multispin-coded algorithm based on
optimizer can exhibit (limited) quantum speedup at all
Refs. [40, 41]. This algorithm performs updates on 64 copies
[36], although there are promising indications from sim-
in parallel, updating all at once. For the r = 7 simulations
ulation[20]andexperimentsonspinglassmaterials[37].
we use a code optimized for bipartite lattices [42]. Imple-
Experimental tests will thus be important. We chose to mentations of the simulated annealing codes are available in
focus here on the benchmark problem of random zero- Ref. [42]. We used the code an ms r1 nf for r = 1, the code
field Ising problems parametrized by the range of cou- an ms r3 nfforr=3andthecodean ss ge nf bp forr=7.
plings. We did not find evidence of limited quantum
speedup for the DW2 relative to simulated annealing in Quantum annealing. To perform quantum annealing one
ourparticularbenchmarksetwhenweconsideredthera- maps the Ising variables σiz to Pauli z-matrices and adds a
transverse magnetic field in the x-direction to induce quan-
tio of quantiles of time to solution, which is the relevant
tumfluctuations,thusobtainingthetime-dependentquantum
quantityfortheperformanceofadeviceasanoptimizer.
Hamiltonian
We note that random spin glass problems, while an in-
terestingandimportantphysicsproblem,maynotbethe H(t)=−A(t)(cid:88)σx+B(t)H , t∈[0,t ] . (9)
i Ising a
most relevant benchmark for practical applications, for
i
which other benchmarks may have to be studied.
The annealing schedule starts at time t = 0 with just the
When we focus on subsets of problem instances in an
transversefieldterm(i.e.,B(0)=0)andA(0)(cid:29)k T,where
instance-by-instance comparison, we observe a possibil- B
T is the temperature, which is kept constant. The system is
ity for a limited quantum speedup for a fraction of the
theninasimplequantumstatewith(toanexcellentapprox-
instances [38]. However, since the DW2 runs at a subop- imation) all spins aligned in the x direction, corresponding
timalannealingtimeformostofthecorrespondingprob- to a uniform superposition over all 2N computational basis
lem instances, the observed speedup may be an artifact states (products of eigenstates of the σz). During the
i
of attempting to solve the smaller problem sizes using annealing process the problem Hamiltonian magnitude
an excessively long annealing time. This difficulty can B(t) is increased and the transverse field A(t) is decreased,
only be overcome by fixing the issue of suboptimal an- ending with A(ta) = 0, and couplings much larger than
the temperature: B(t )max(max |J |,max |h |) (cid:29) k T.
nealing times, e.g., by finding problem classes for which a ij ij i i B
At this point the system will again be trapped in a local
the annealing time is demonstrably already optimal.
minimum, and by repeating the process one may hope to
There are several candidate explanations for the ab-
find the global minimum. Quantum annealing can be viewed
sence of a clear quantum speedup in our tests. Perhaps as a finite-temperature variant of the adiabatic quantum
quantum annealing simply does not provide any advan- algorithm [8].
tages over simulated (quantum) annealing or other clas-