Internal Robustness: systematic search for systematic bias in SN Ia data

2.9 MB·English

by  Luca Amendola

Internal Robustness: systematic search for systematic bias in SN Ia data
Luca Amendola,1 Valerio Marra,1 and Miguel Quartin2
1Institut für Theoretische Physik, Universität Heidelberg, Philosophenweg 16, 69120 Heidelberg, Germany
2Instituto de Física, Universidade Federal do Rio de Janeiro, CEP 21941-972, Rio de Janeiro, RJ, Brazil
A great deal of effort is currently being devoted to understanding, estimating and removing
systematicerrorsincosmologicaldata. IntheparticularcaseoftypeIasupernovae,systematicsare
startingtodominatetheerrorbudget. HereweproposeaBayesiantoolforcarryingoutasystematic
search for systematic contamination. This serves as an extension to the standard goodness-of-fit
tests and allows not only to cross-check raw or processed data for the presence of systematics but
alsotopin-pointthedatathataremostlikelycontaminated. Wesuccessfullytestourtoolwithmock
cataloguesandconcludethattheUnion2.1datadonotpossessasignificantamountofsystematics.
Finally, we show that if one includes in Union2.1 the supernovae that originally failed the quality
2
cuts, our tool signals the presence of systematics at over 3.8-σ confidence level.
1
0
2 PACSnumbers: 98.80.Es,97.60.Bw,02.50.Cw
Keywords: ObservationalCosmology,Supernovae,Probabilitytheory
p
e
S I. INTRODUCTION passes)andthefluxreferenceofsuchfilters(forareview,
0 see for instance [22]). On the other side, non standard
1 cosmological models might affect our parameter estima-
The best evidence for the accelerated expansion of the
tion: for instance, any anisotropy in the expansion rate
universe still comes, after 15 years from the earliest re-
] wouldshowupasananisotropyintheSNIacosmological
O sults [1, 2], from the supernovae Ia (SNIa). There are
parameters. Therehavebeenofcoursemanysearchesfor
now several hundreds SNIa useful for cosmological pur-
C
such systematic biases. All of them, however, assume a
poses, ranging in distance up to z ≈1.7. The SNIa have
h. been compiled in different datasets [3–8], taking into ac- specific effect (say, gravitational lensing [23] or cosmo-
p count different sets of possible systematics and making logical anisotropy [24–26]) and test whether this effect is
- enough to make some SNIa incompatible with the oth-
o use of two different approaches to standardize these pri-
ers. In other words, one proceeds by testing a specific
r mordial candles [9, 10]. Nevertheless, every analysis per-
t prejudice.
s formed on these datasets confirms that a present cosmic
a acceleration explains satisfactorily the data. The same In this paper we propose an alternative approach. We
[ conclusionisnowsupportedalsobyseveralotherlinesof wish to perform a systematic search of biases without
1 evidence, such as measurements of the Baryonic Acous- having any preferred selection criteria. In other words,
v tic Oscillations (BAO) [11, 12], of the anisotropies of the we try to answer the following question: is there any
7 CosmicBackgroundRadiation(CMB)[13]andoftheage subsetofSNIathatisstatisticallyincompatiblewiththe
9 of the oldest stars known [14–16]. others? That is, is there a subset of SNIa that could
8
The recent increase in the number of observed super- be described by parameters which are incompatible with
1
novae is also driving a huge effort to understand and those that describe the other SNIa? In a sense, this is a
.
9 control possible sources of systematics that may under- directgeneralizationofthesearchforoutliers. Insteadof
0 mine the progress in the cosmological interpretation. A searchingforsingleoutliers,i.e.SNIathatappearstatisti-
2
recent analysis [8] claims indeed the systematic uncer- callyincompatiblewiththeothers(say,someparameters
1
: tainties are already larger than the statistical ones and that describe their light curves are too far off from the
v theissuewillbemuchmoreimportantinthenearfuture othersortheirdistancemodulijustendupveryfarfrom
Xi as we forecast an increase in the number of observed su- the overall Hubble diagram), we search for subsets of,
pernovae by 1 order of magnitude in the next ∼ 5 years say,dozensofSNIaatoncewhoseparametersareincom-
r
a (forexample,withtheDarkEnergySurvey[17])andby2 patiblewiththeothers. AswillbeshowninSectionIIB,
orders of magnitude in the next ∼15 (for example, with theproposedgeneralizationreducestothestandardout-
the Large Synaptic Survey Telescope [18]). It is thus liersearchinthelimitinwhichthewholedataisdivided
necessary to continue investigating the SNIa datasets in into two complementary datasets, one of which contains
search of such systematic effects and of additional cos- a single element.
mological information. On one side, in fact, we are al- The standard tool to compare whether a particu-
ready aware of many effects that could come into play lar dataset is compatible with a proposed model is
to alter the SNIa apparent magnitude: contamination the goodness-of-fit test, which gives well defined prob-
from non-Ia supernovae, dust absorption in both host ability statements about such agreement. However, the
galaxy and Milky Way, gravitational lensing distortions, information obtained is limited, and this simple analysis
local velocity flows et cetera; not to count systematics mayhideproblemsinbothdataandmodel. Forinstance,
which arise from selection effects [19–21] and from rely- if a given model parameter affects only a small fraction
ing solely on photometry (typically in just a few band- of the data, even if such parameter turns out to disagree
2
with this fraction the goodness-of-fit may still claim an to write
overall good fit. To address this issue, a modification
P(M)
of the test, dubbed parameter goodness-of-fit, was pro- L(M;x)=E(x;M) , (2)
P(x)
posed in [27] and later extended in [28]. The parameter
goodness-of-fit method nevertheless still relies on com-
i.e. the posterior probability L of having model M given
parisonsofχ2 valueswhichareonlysensitivetothelocal
the data. We can finally use the latter equation to com-
minimum, and not to the entire likelihood. Here, in-
pare quantitatively two models taking the ratio of their
stead,wewilladoptafullyBayesianapproachsoastouse
probabilities (so that P(x) cancels out):
all the information available, e.g. a possible overlapping
of the likelihoods surfaces. We dub internal robustness L(M ;x) P(M )
thefundamentalquantityevaluatedinthismethod. The L(M1;x) =B12P(M1), (3)
2 2
nameismotivatedbytheanalogousquantityrobustness,
which was recently defined in a context in which the two whereweintroducedtheBayesratio(sometimesreferred
datasets refer to different observational probes [29] (and to as Bayes factor)
which we will henceforth refer to as external robustness,
for differentiation). In particular, Ref. [29] showed that B = E(x;M1). (4)
the robustness is an estimator “orthogonal” to the Fig- 12 E(x;M )
2
ureofMerit,whichissensitivetotherelativeorientation
of the two probes but not to the distance between the Often, however, one assumes that P(M1) = P(M2) and
two-experiment confidence regions. we adopt this choice here. A Bayes ratio B12 > 1 (< 1)
saysthatcurrentdatafavorsthemodelM (M ). Aswe
This paper is organized as follows. In Section II we 1 2
willseeinthenextSectiontheBayesratiowillbecentral
will introduce the formalism of the internal robustness.
in the definition of internal robustness.
In Section III we will describe how we will systemati-
Supposenowthelikelihoodisgaussianinthedatawith
cally search for bias in SNe data. In Section IV we will
covariance matrix Σ and expected means m . Then
show the results relative to a biased test catalogue, the i
Union2.1 catalogue augmented with the supernovae that
did not pass the quality cuts, and the actual Union2.1 L = (2π)−N/2|Σ|−1/2e−21χ2, (5)
dataset. Aswewillsee,ourmethodwillbeabletodetect
where the χ2 is defined as:
thesystematicbiasinthefirsttwocatalogues. Moreover,
our analysis does not show signs of systematic effects in χ2 ≡(x −m )tΣ−1(x −m ). (6)
i i ij j j
the actual Union2.1 catalogue. Finally, we will give our
conclusions in Section V, and explain some technical de- The best-fit (minimum) χ2 is then
tails in Appendix A.
We will adopt the following notation. Bold face will χb2 = (xi−mbi)tΣ−ij1(xj −mbj), (7)
distinguish a vector x or matrix A from their compo-
where m are the best-fit means. The maximum of the
nentsx andA ,asuperscripttwilldenoteatransposed bi
i ij
likelihood is then:
vector or matrix, |A| will represent the determinant of a
mcoartrreisxpoAn,dainngdqauhaanttiwtyi.llindicatethebest-fitvalueofthe Lmax = (2π)−N/2|Σ|−1/2e−21χb2, (8)
so that we can rewrite Eq. (5) as
II. FORMALISM L = Lmaxe−12(χ2−χb2). (9)
According to (1), the N means m depend on n parame-
i
A. Bayesian evidence and its Fisher approximation tersθ ,i.e.m =m (θ).1 Thebest-fitvaluesm arethen
k i i bi
(cid:0) (cid:1)
functions of the best fit estimators θbk, i.e. mbi = mi θb .
Let us first of all recall some statistical definitions in HereforsimplicityweassumethatΣdoesnotdependon
the Bayesian context [30, 31]. The Bayesian evidence is the parameters, but this assumption can be easily lifted.
defined as Let us assume now that the likelihood can be approx-
ˆ imated near θb by a Gaussian distributions also in the
E(x;M)= L(x;θM)P(θM)dnθM, (1) parameters, i.e.
L(x;θ) ’ f(x;θ) ≡ Lmaxe−21(θi−bθi)tLij(θj−bθj), (10)
where x = (x ,x ,...,x ) are N random data, θM =
1 2 N
(θ ,θ ,...,θ ) are n theoretical parameters that describe
1 2 n
the model M, L is the likelihood function, and P is the
prior probability of the parameters θM. If P(M) is the 1 Notethattosimplifythenotationwewilldropinthefollowing
prioronaparticularmodelM,wecanuseBayes’theorem equationsthesuperscriptM ofthemodelinquestion.
3
where L in the exponential factor is the inverse of the M . Inthiscasethetotalevidencecanbewrittenasthe
ij C
covariancematrixofthelikelihood(orFishermatrix,see product of the individual evidences:
e.g.[31,32])andwherenowthedataareinsidethebest-
fit estimators θbi. Similarly, we assume a gaussian prior Eind =E1 E2. (17)
so that
We can now use the Bayes ratio of Eq. (4) to quantify
|P|1/2 which hypothesis is favored. We thus compute2
P(θk)= (2π)n/2 e−12(θi−eθi)tPij(θj−eθj), (11)
E E(x;M )
B = tot = C , (18)
tot,ind E E(x ;M )E(x ;M )
where θei are the prior means and P is the prior matrix. ind 1 C 2 S
It is now possible to evaluate the evidence analytically.
and define
Using the relation
ˆ
R ≡ logB (19)
(2π)n/2 tot,ind
dnxe−21xtAx+vtx = e12vtA−1v (12)
|A|1/2 as the internal robustness. As discussed in the Introduc-
tion this quantity is related to the external robustness
in
ˆ originally defined simply as robustness in [29].
The previous equations give the general definition of
E = f(x;θ)P(θ)dnθ
internal robustness. It is however useful to evaluate ana-
ˆ
|P|1/2 (cid:20) 1 lytically R in the Fisher approximation. Using Eq. (15)
= Lmax(2π)n/2 exp −2(θi−θbi)tLij(θj −θbj) one finds:
1 (cid:21) (cid:18)|L +P ||L +P |(cid:19)1/2 Ltot
− 2(θi−θei)tPij(θj −θej) dnθ, (13) Btot,ind = |1Ltot+CPC2||PS|S L[m1]axmLax[m2]ax
h P (cid:0) (cid:1) i
one finds ×e−12 θbttotLtotθbtot−θt0totFtotθt0ot− i θbitLiθbi−θi0tFiθi0 −θe2tPSθe2
E =Lmax ||PF||11//22 e−21(cid:0)θb−θe(cid:1)tLF−1P(cid:0)θb−θe(cid:1), (14) ’ (cid:18) |L1||L2| (cid:19)1/2 (cid:18)|Σ1||Σ2|(cid:19)1/2 e−12(χb2tot−χb21−χb22),
|L ||P | |Σ |
tot S tot
(20)
where F = P +L. It will be convenient to rewrite the
whereinthebottomlinewesimplifiedusingEq.(16),i.e.,
above equation as
assuming the prior to be much broader than the likeli-
(cid:16) (cid:17) hoods. Notice that the (2π)−N/2 factors cancel out since
E = Lmax ||PF||11//22 e−21 θbtLθb+θetPθe−θ0tFθ0 , (15) Nfatrotw=e hNa1v+e Nas2s,uwmheedregaNuissisiatnhietysiozfetohfethliekesulibhsoeotddsi,.tShoe
existenceoftwoindependentdistributionsandtheuseof
where θ0 = F−1(cid:0)Lθb+Pθe(cid:1). In the limit in which the averybroadprior. Ifwemaketheadditionalassumption
prior is very broad (so that Pij (cid:28) Lij and therefore that the data points themselves are independent of one
Fij →Lij)theargumentoftheexponentialvanishesand another (henceforth referred to as “raw data”), then the
we have simply covariance matrix is diagonal, one has |Σ | = |Σ ||Σ |
tot 1 2
and we get the final formula for the internal robustness
|P|1/2
E ’ (2π)−N/2e−21χb2 . (16) in the Fisher approximation:
|L|1/2|Σ|1/2
(cid:18) (cid:19)
R=R + 1log |L1||L2| − 1(cid:0)χ2 −χ2−χ2(cid:1), (21)
0 2 |L | 2 btot b1 b2
tot
B. Definition of internal robustness
where R is a constant coming from the unknown deter-
0
Eq. (1) and its Fisher approximation Eq. (15) allow minant of the systematic prior, R0 =−12log|PS|.
to compute the evidence E for a given cosmological ThefirstfactorinEq.(21),formedoutofthedetermi-
tot
model M and dataset d . Suppose now that the data nants, expresses Occam’s razor factor of parameter vol-
C tot
are actually coming from two completely different, and umes, while the second penalizes R if the two probes are
therefore independent, distributions. In other words, as- very different from each other (so the hypothesis that
sume that a subset d of d actually depends on a to- they come from different models, or equivalently that
2 tot
tally different set of parameters (say, the properties of
the SNIa progenitors or galaxy environment), i.e., it is
fully described by the systematic parameters of model
M . Contrarily, the complementary set d =d −d is 2 It is straightforward to generalize Eq. (18) to more than two
S 1 tot 2
still described by the cosmological parameters of model partitionssuchthatEind=E1E2E3...
4
systematicsareimportant,isfavored). Weexpect,there- of d is constrained within a fixed realization of d , on
2 tot
fore,theinternalrobustnesstobeameasureofhowmuch which the iR-PDF depends. As a simple example of the
subsets of a dataset overlap: the more they do, the more difference between the latter two distributions, the iR-
compatible the two datasets are. PDFhasacompactanddiscretesupportasitissampled
In order to help intuition, we can evaluate R for the over a finite number of subsets. We will see in Section
simplifiedcaseinwhich a)wecanneglectthelogarithmic IVA4, however, that for the datasets treated in this pa-
part; b)thereisonlyoneparameter; c)theerrorsareall perthebinnediR-PDF(neglectingOccam’srazorfactor)
identical (σ = σ); and d) the subset d consists of a is rather close to a χ2 distribution with n degrees of
i 2 tot
single point x . Then we have χ2 = 0 and for N ’ freedom.
2 b2 1
Ntot (cid:29)1 TheiR-PDFgivestheprobabilitythatagivenvalueof
R is realized among the available subsets. However, dif-
1(x −m )2
R≈− 2 btot (22) ferentlyfromtheeR-PDF,theiR-PDFcannotbeusedto
2 σ2 assessthesignificanceofagivenvalueofR. Todoso,we
needtocomputeadistributionofiR-PDFs,whichwewill
i.e. R reduces to the scatter of x from the best fit m
2 btot
obtain by evaluating the robustness in many Montecarlo
evaluated by fitting the remaining N elements. There-
1
realizations of mock catalogues.
fore a large and negative R means x is an outlier.
2
C. Statistical properties of internal robustness D. Systematic parameters
In order to understand the statistical properties of the Generally speaking, there are two possible choices for
internal robustness let us start by fixing the subset d2 the parameters for the subset d2. In the first case,
to some d∗2 (d∗1 is just the complementary subset). Let the parametrization is analog to the cosmological one,
us also assume that data come from the cosmological e.g. {Ω ,Ω ,α}, where Ω ,Ω are the present-day
m0 Λ0 m0 Λ0
model MC only. From Eq. (21) we can then calculate matteranddark-energyparametersandαiscombination
the probability distribution function of R(d∗2) (denoted of the unknown magnitude offset M0 (sum of the SNe
as eR-PDF). If one also neglects the logarithmic term absolute magnitudes, of k-corrections and other possi-
(Occam’srazorfactor),then(inthisveryparticularcase) ble systematics) and the Hubble parameter H : α ≡
0
Rbecomestheparametergoodness-of-fittest[27],andit M +25−5log H [31]. In this case P = P . This
0 10 0 S C
was shown [28] that R(d∗2) is distributed as a χ2 distri- would be the preferred choice if we expect some of the
butionwithntot degreesoffreedom(d.o.f.). ThefulleR- SNe to be better described by different cosmological pa-
PDF is then a modified χ2 distribution with ntot d.o.f., rameters. For instance the parameters could differ in
not too distorted as Occam’s razor factor is logarithmi- different line-of-sight directions (say, 3 different Hubble
cally suppressed. If we now drop the assumption that parameters H , H and H , as in anisotropic mod-
x0 y0 z0
data come from the cosmological model MC, we can use els with shear [26, 33]) or in different redshift shells (say,
the(fiducial)eR-PDFtoassessthesignificanceofagiven Ωin andΩoutasinsomeinhomogeneousmodels[34–36]).
m0 m0
value of R(d∗2), for example of a low value that could in- Wecouldalsoerroneouslyinterprettheunknownsystem-
dicatethatthedatasetissystematicsdriven. Weremind atic parameters as the cosmological ones and therefore
indeed that in our fully Bayesian context the robustness employ the same parameter names and the same prior
R is related to the Bayes ratio of the evidences and a function.
small (large) R disfavors (favors) the description of the
In the second case, not to be explored here, one could
dataset by the cosmological model M alone.
C choose a completely phenomenological parametrization
Asfarastheinternalrobustnessisconcerned,however,
such as
we do not intend to fix d to a particular subset (even if
2
we may still do so if useful). The above way of proceed- j
X
ing is suited indeed to the external robustness, the aim m(z)= λ f (z) (23)
i i
of which is to analyze two different datasets (e.g. CMB i=0
and BAO). The idea behind the internal robustness is
insteadtokeepthetotaldatasetfixed(i.e.nottobecon- for the theoretical magnitudes. Here the functions f(z)
cerned with the statistical distribution of R(dtot)) and could be arbitrarily chosen, e.g. fi(z) = zi. If j = 2
evaluateR(d2)forallthepossiblepartitionsofdtot, thus we have then m = λ0 +λ1z +λ2z2. Since the λ’s are
generating a distribution which we call iR-PDF. Inter- linearparameters, bothbestfit andFishermatrixwould
nal and external robustness have therefore very different beanalytical. Thissecondchoicewouldbeappropriateif
statistical properties. oneexpectssomeoftheSNetobedominatedbysystem-
The fiducial iR-PDF (assuming that data come from aticeffectsunrelatedtocosmology,saybecauseofsample
the cosmological model M only) is a highly nontrivial contamination or strong environmental effects.
C
object. Even if one neglects Occam’s razor factor, it is Restrictingourselvestothefirstcase,wecanmarginal-
not a χ2 distribution with n d.o.f., as the sampling ize over α analytically [31]. Let us define the sums (re-
tot
5
member that the covariance matrix Σ is diagonal) Note that in the following results we will express the
numerical values of the robustness as R−R −log(2π),
XN0 Mn to which we will refer simply as R. 0
S = i , (24)
n σ2 In Section IV we shall use supernova data provided
i=1 i by the Union2.1 [8] collaboration in the form of a 3-
where N0 = N1,2, Mi ≡ xi −mi = xi −5log10d¯L(zi), columnmatrix,eachrow{zi,xi,σi}consistingofredshift
d¯ = d H /10pc and d is the luminosity distance. zi, distance modulus xi and distance modulus error σi.
L L 0 L
ThismatrixwascomputedusingtheSALT2method[10].
Marginalizing over the constant offset α we obtain
Used in this way, the supernova data are rigorously not
−logL= 21χ2mar+ 12log2Sπ0 +XN0 log(cid:0)√2πσi(cid:1), (25) ionbdteapineenddeanftt,erapsrtohceessvianlguetsheofratwheddaitsataanscseumminogdualipaarre-
i=1 ticularcosmologicalmodel(see,e.g.[37]). Asinthisfirst
where work we mainly aim at presenting the method, we will
ignorehoweversuchcorrelations.3 Note,however,thatif
S2
χ2 =S − 1 . (26) on one hand this is a caveat for the results that follow,
mar 2 S0 on the other hand it presents an opportunity to cross-
As the likelihood is gaussian in the data, minimizing the check the Union2.1 data (often naively employed in such
marginalized χ2 with respect to {Ω ,Ω } gives the concise form) for leftover systematics.
mar m0 Λ0
same result as minimizing the original χ2 with respect
to {Ω ,Ω ,α}. Therefore, even though we are in a
m0 Λ0
Bayesian context, χ2 is still distributed as a χ2 with III. SCANNING THE SUBSETS
bmar
N0−3 degrees of freedom. Note also that if the original,
non-marginalizeddataisindependent,thensowillbethe InordertoobtainthefulliR-PDFwehavetocompute
data marginalized over α. To see this, one can rewrite χ2 and L for every partition d of a given dataset
b1,2 1,2 1,2
the marginalized likelihood as of N elements. There are
tot
(cid:20) (cid:21)
1(cid:0) (cid:1)t −1 (cid:0) (cid:1) 2Ntot−1−1≈100.3Ntot (31)
L∝exp − x−m Σ x−m , (27)
2
possible partitions and we find that in the present ap-
where Σ−1 = Σ−1−Σ−2/S , which for a diagonal Σ is plication a complete scan of all subsets is unfeasible for
0
still a diagonal matrix. N (cid:38) 20. The issue then arises of which subset Ξ
tot
Recalling that in general a Fisher matrix is given by among all possible partitions to form. We extract from
the entire Union2.1 catalogue a number T of subsets d
∂2 logL 2
Lpq =− ∂θ ∂θ , (28) composed by a number N2 of SNe between N2,min and
p q N chosen at random among all the possible combi-
2,max
nations. However, a pure random sampling (i.e. uniform
we get for the cosmological Fisher matrices L , L and
tot 1
inthespaceofallpossiblepartitions)wouldpickwithex-
L the general expression
2
tremelyhighprobabilityonlythemostpopulatedsubsets
1 1 (cid:0) (cid:1) (in our case with N ∼ N ). Since we would like to
L = S − S S +S S (29) 2 2,max
pq 2 2,pq S0 1 1,pq 1,p 1,q explore also the smaller subsets, we adjust the selection
(cid:18) (cid:19)
= XMi,pq M − S1 +XMi,pMi,q so as to obtain a distribution approximately uniform in
i σi2 i S0 i σi2 Nva2lu(ei.eo.faNpp)r.oxWimeatceallyl tehqiusaplanrutmicbuelarrosfestubΞs(eTts),foarndevtehrye
2
− 1 XMi,p XMj,q following analysis depends on it. In particular T gives
S0 i σi2 j σj2 the statistics of the analysis, while the definition of Ξ
determines the way the sets have been chosen. We will
= 5 X 1 (cid:20)d¯Li,pd¯Li,q − d¯Li,pq(cid:21)(cid:18)M − S1(cid:19) consider different strategies in forthcoming work.
ln10 i σi2 d¯2Li d¯Li i S0 The upper limit we use is N2,max = Ntot/2. This
+ (cid:20) 5 (cid:21)2"Xd¯Li,pd¯Li,q − 1 X d¯Li,p X d¯Lj,q #. ilsogdicuael ptoaratmheetfraizcattitohna,t,thaesrowbeusatrneesussiisngsytmhme ectorsicmoin-
ln10 i σi2d¯2Li S0 i σi2d¯Li j σj2d¯Lj the datasets d1,d2 so that scanning half of the cata-
logue is enough. In order to discuss N it is impor-
2,min
Finally, the robustness in the Fisher approximation is
tant to stress that we consider a much larger parameter
given in this case [see Eq. (25)] by:
1
R = R − log(2π) (30)
0 2
(cid:18) (cid:19) 3 It is worth stressing that the important factor is the indepen-
+ 1log S0,1S0,2|L1||L2| − 1(cid:0)χ2 −χ2−χ2(cid:1). dence of the data points, not of their error bars, which even if
2 S0,tot |Ltot| 2 btot b1 b2 completelycorrelatedwouldnotaffecttheresults.
6
intherobustnesscomputationatalevel(cid:46)0.1. Wefound
that the value N =90 satisfies this requirement.
2,med
IV. RESULTS
InanalyzingtheUnion2.1catalogueof580SNewewill
restrict to the case of the curved ΛCDM model, i.e., we
allow for spatial curvature but fix the equation of state
parameter (w = −1). The cosmological parameters are
therefore the present-day density parameters Ω and
m0
Ω , with the addition of the nuisance offset α. We will
Λ0
consider other parameterizations in forthcoming work.
Our results will be divided into three parts. First
in Section IVA we will test our method with a mock
dataset whose SNe were drawn from two very different
cosmologicalmodels. InSectionIVBwewillanalyzethe
Figure 1. Comparison between the exact robustness R and Union2.1dataset[8]includingpreviously-excludedsuper-
E
itsFisherapproximationR forarandomsetofsubsets. Each novae(i.e.,SNIathatdidnotpassallthequalityselection
F
subsetisrepresentedbyapoint,whichiscolorcodedaccord- cuts); this also should be a test for our method. Finally,
ingtothecorrespondingnumberN2 ofSNeinthesubsetd2. in Section IVC we will present our results regarding the
This analysis clearly shows that, as far as the robustness is actual Union2.1 dataset.
concerned, the Fisher approximation holds better for larger
Before dealing with our results, it is useful to define
subset sizes. See Section III for more details.
what we mean by “mock catalogue”. A mock catalogue
M for a given dataset D is a synthetic unbiased dataset
generated using the best-fit model of D as the fiducial
model. Moreprecisely, wekeepfixedtheredshiftsz and
i
space than the usual physical one, as Ω and Ω also
m0 Λ0 errorsσ ofD,andchangethedistancemodulitox =
i mock
parametrize the (possibly cosmology unrelated) system-
m +x wherex isdrawnfromagaussian
fiducial random random
aticparameters. Therangeweadoptis−10<Ω <10
m0 distribution of zero mean and standard deviation σ .
i
and −20 < Ω < 10, and we exclude the {Ω ,Ω }
Λ0 m0 Λ0
region of the parameter space for which the expansion
rate H(z) is negative for z < 2, which well accommo-
dates the redshift range of the Union2.1 dataset. This A. Systematics-driven dataset
is a relaxation of the usual no-big-bang excluded region,
see Appendix A for more details. The value of N 1. Dataset and iR-PDF computation
2,min
is then found by demanding that the likelihood of the
smallersubsetd2hassupportwithintheparameterspace In order to test our method we have generated a
considered. We have found empirically that a value of systematics-driven dataset D in the following way.
EdS
N2,min =10 satisfies on average this requirement. First we have created a mock catalogue of Union2.1.
We will now explain how we actually computed the Thenwereplaced100randomlychosendistance-modulus
robustness. For subsets with N larger than a certain entries with others drawn taking the Einstein-de Sitter
2
N we have found that the likelihood can be ade- (EdS)model(flat,matter-dominated)asthefiducialone
2,med
quatelyrepresentedbyaGaussiandistributioninthepa- (instead of the best-fit model of Union2.1). These 100
rameter space, so as to legitimate the Fisher approach. SNeareshowninredinFig.2. OneexpectsforD very
EdS
The advantage of using the Fisher matrix is in the com- lowrobustnessvalues,asforthesubsetof100EdSSNeit
putational speed gain, as only the maximum of the like- isclearlyfavoredthepossibilitythatd andd haveinde-
1 2
lihood and few derivatives have to be found numerically. pendent cosmological parameters (which, we remind, we
ForsmallersubsetsN ≤N ,however,thelikelihood use to parametrize also the systematics). In the case of
2 2,med
deviates from a Gaussian distribution and we are forced d being exactly the subset of 100 EdS SNe, one obtains
2
to integrate the likelihood numerically over the full pa- theplotofFig.3whichshowsthe1,2and3σconfidence-
rameter space. In order to empirically find N we levelcontoursford andd ,d independently. Thecon-
2,med tot 1 2
havecomputedboththeexactrobustnessR ofEq.(18) toursareindeedfarapartandtherobustnessisextremely
E
and its Fisher approximation R of Eq. (21) for a ran- low, R ’−97.5 (as previously mentioned, we know a
F min
dom set of subsets. Fig. 1 shows how the discrepancy posteriori that R is typically a O(1) quantity). Fig. 3
decreases as the subset d becomes larger. As we will summarizes nicely the ultimate goal of internal robust-
2
seebelowinFig.4,theiR-PDFvariesonascaleoforder ness, that is, to go from the original set in green to the
unityinrobustness. Therefore,wewanttokeeptheerror “decontaminated” set in red. The contours do not look
7
46
mag333444468024 (cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:224)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224) (cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224) (cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230) (cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224) (cid:87)0(cid:76)(cid:45)0001....5050
(cid:45)1.0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4
z
(cid:45)1.5
Figure 2. Plotted is the distance modulus of the SNe of
the systematics-driven dataset DEdS in which 100 SNe have 0.0 0.5 1.0 1.5 2.0
beendrawnfromtheEdSmodelasfiducialmodel(reddata) (cid:87)
m0
andtheremainingfromthebest-fitmodelofUnion2.1(blue
data). Without the different coloring it would be difficult to Figure3. 1,2and3σ confidence-levelcontoursforsubsetsd
1
spot by eye the EdS SNe (the challenged reader could try it (blue)andd (red),whered ismadeofthe100systematics-
2 2
on a B&W printer). See Section IVA1 for more details. driven EdS SNe shown in red in Fig. 2. The contours are far
apartandtherobustnessisindeedextremelylow,R’−97.6.
Also shown in green are the confidence-level contours for the
complete dataset d . The 100 EdS SNe clearly bias the
tot
significantly different (i.e. same precision), but their po- likelihood surface of the full dataset. See Section IVA1 for
sition has substantially moved by ∼3σ (i.e. much better more details.
accuracy).
We will now pretend that D is made of real data
EdS
and test the sensitivity of our method with it. Before
proceeding, however, one can see that the dataset is sick
by means of the standard goodness-of-fit test given by tobeaddedtoσ inordertohaveχ¯2 =1. ForD one
int EdS
χ¯2 = χb2/(Ntot −3) (the cosmological model has three needsσienxttra =0.0356magnitudes;thatis,toincreasethe
parameters). ForDEdS wefindindeedχ¯2 =1.39, thatis, errors. As a consequence the contours of Fig. 3 become
the catalogue is incompatible with the theoretical model slightly broader and the minimum robustness is larger,
at 6-σ level. Nevertheless we advocate that internal ro- even though still extremely low: R ’−70.9.
min
bustness is a better test than the standard goodness-of-
fit;inotherwordswewouldliketoshowthatwecangive Finally, we computed the internal robustness for the
a stronger exclusion. In order to do so, we will “normal- (normalized) D dataset using the set of partitions
EdS
ize” D by adding a constant σextra to the errors σ Ξ(T) with a statistics of T = 5·105 subsets (see Sec-
EdS int i
such that χ¯2 = 1. The idea is that the normalized cata- tion III). As explained in Section IIC, in order to deter-
logue passes the goodness-of-fit test but not the internal mine if the iR-PDF of D passes or not the robustness
EdS
robustness one. test,wehavetogenerateadistributionofiR-PDFs,which
The technique of normalizing a catalogue to χ¯2 =1 is we obtain by computing the internal robustness for 100
actuallyoftenusedinsupernovacosmology[38]. SNeare mocks M . Each mock iR-PDF is generated using the
j
indeed imperfect standard candles with a residual scat- same set prescription Ξ(T) but with a lower statistics of
ter, called the intrinsic dispersion, of roughly σ ∼ 0.1 T =5·104,asthefluctuationsamongthemocksaremore
int
magnitudes. As SN physics is not yet thoroughly un- important than the “sampling” fluctuations due to pois-
derstood, σ is not tightly constrained, and it is often sonian errors. The mock catalogues M have been also
int j
determined by demanding that χ¯2 =1.4 SNe catalogues normalized to χ¯2 = 1. It is worth saying at this point
are therefore perfect candidates for testing the internal that the robustness test is quite expensive from a com-
robustness method. Since the Union2.1 data already in- putational point of view. The numerical results of this
clude implicitly a σ , what we call σextra is the amount paper have been obtained with Wolfram Mathematica
int int
8 and the average CPU time to calculate the robustness
valueofagivenpartitionwas∼2−3seconds(luckilythe
computationiseasilyparallelized). ThesizeofT andthe
4 Notethatthisproceduremayhideproblemswiththetheoretical numberofmocksusedarethereforethemainconstraints
model being used, as it is shown by the very example of DEdS. in the final results: the higher the statistics, the clearer
Forfurtherdiscussionandalternativessee[39–41]. the signal one may get.
8
0.35 8
e
0.30 DEdS nc
e 6
r
e
0.25 ff
di
e 4
DF0.20 ativ
P el 2
0.15 r
d
e
z
0.10 ali 0
m
r
o
0.05 n(cid:45)2
0.00
(cid:45)4 (cid:45)2 0 2 4 (cid:45)20 (cid:45)15 (cid:45)10 (cid:45)5 0 5
Robustness Robustness
Figure 4. [Left]: binned internal-robustness PDF (orange curve) for the dataset D of Fig. 2 against σ-bands (gray areas)
EdS
from (unbiased) mock catalogues. [Right]: same as the left panel for larger bins (dashed lines) and lower robustness values.
Moreover, the bin height values h have been translated and scaled according to h → (h −h¯ )/σ in order to uniformly
k k k k hk
show the signal across the various bins. One can clearly see that while the body of the iR-PDF is compatible with the mocks,
the low-robustness tail is detected as being driven by systematics. The D datum relative to the bin (−20,−6] lies at 4.2-σ
EdS
confidence levels. See Section IVA2 for more details.
2. Analysis of the iR-PDF body for which h¯ /σ (cid:29) 1, but not in the tail as il-
k hk
lustrated in Fig. 5 where the distributions for two low-
robustness bins are shown. The reason is that the iR-
The left panel of Fig. 4 shows the iR-PDF of D
EdS
PDFsofthemockshaveacompactsupportwhichhavea
(solid orange line), which has been obtained by binning
variable R . At a given low value R in the tail, there-
therobustnessvalueswithinN binsofwidths∆R . The min
b k
fore, some of the robustness distributions of the mocks
samebinningprescriptionhasbeenusedtocalculatealso
will be very close (if not identical) to zero, with the con-
the bin heights h of the unbiased mocks M , which
kj j
have been used to compute mean h¯ and standard de- sequence that the distribution of the bin heights will be
k
skewed, thus deviating from gaussianity. Note that this
viation σ for the N bins. By assuming that the bin
hk b cannot be cured by using a higher statistics T as this
heights within a given bin are distributed according to
a gaussian PDF with mean h¯ and the standard devia- feature is related to the existence of an Rmin for the iR-
k
PDF.
tion σ , we have then drawn the σ-bands bounding a
hk
systematics-free iR-PDF (gray areas in the left panel of In order to analyze the tail, the standard way to pro-
Fig.4). FromthisplotitseemsasiftheiR-PDFofD ceed would be to generate many mock iR-PDFs so as to
EdS
passes the internal robustness test. computethenon-gaussiandistributioninthebinheights
Oneexpects,however,thesignaltobeconcentratedin numerically. As remarked earlier, however, it is numer-
thelow-robustnesstailofthePDF,inwhichsystematics- ically expensive to obtain an iR-PDF and we limit our
driven SNe should dominate one of the two partitions sample of mock catalogues to 100. In order to properly
d . ThespuriousiR-PDFcanindeedbelooselythought quantify the signal we have then proceeded in two ways.
1,2
as the sum of a systematics-free PDF and a systematics- The first and most obvious is to check if any mock has a
driven perturbation. The systematics-free PDF will be- bin height larger than DEdS. As shown by Fig. 5 this is
have according to the σ-bands, which do not change siz- not the case for the lowest bin (where we expect the sig-
ably if the biased subset is sufficiently small. The per- naltobestrongest),andwecansoconcludethatDEdS is
turbation will then be given by the 100 EdS SNe which, systematics-drivenataconfidencelevelbetterthan99%,
when d coincides with them, will add a single point to or 2.6σ.
2
the histogram at R ’ −70.9. This point is clearly The second way is to use a template to be fitted to
min
not observable in practice as one cannot scan all the the data, and then use the fitted template to assess the
10174 possible subsets. At the expense of a higher ro- significance of the datum relative to D . We used as
EdS
bustness, however, there will be many subsets d with a templatethePearsondistribution[42,43],whichisfound
2
fraction of the 100 EdS SNe, thus generating a stronger by demanding that its first four moments coincide with
low-robustness tail in a biased dataset. themomentsfromthebinheights. Theresultisdepicted
Inordertoanalyzethelow-robustnesstailofthePDF, by a solid black in Fig. 5, where also the gaussian dis-
we have to drop the gaussian assumption. The latter is tribution is plotted for comparison. As one can see the
indeed a good approximation only for the bins in the fittedPearsondistributioncorrectlyreproducestheskew-
9
Robustnessbin:(cid:68)R(cid:61)(cid:72)(cid:45)20,(cid:45)6(cid:68) Robustnessbin:(cid:68)R(cid:61)(cid:72)(cid:45)6,(cid:45)4(cid:68)
80000
2500
2000
60000
F F1500
D D
P40000 P
1000
20000
500
0 0
0.0000 0.00002 0.00004 0.00006 0.00008 0.0001 0.0005 0.0010 0.0015
binheight binheight
Figure 5. Binned distributions of bin heights for the lowest (left panel) and next-to-lowest (right panel) robustness bins of the
mock robustness distributions used to make the σ-bands shown on the right panel of Fig. 4. The bin height corresponding to
D is shown as an orange vertical line, and corresponds to the orange curve on the right panel of Fig. 4. The dotted line is
EdS
afittedgaussiandistribution. ThesolidlineisthefittedPearsondistributionwhichhasbeenusedtodrawtheσ-bandsshown
on the right panel of Fig. 4. See Section IVA2 for more details.
nessofthedata,andinparticulardoesnotgotonegative 0.0020
bin height, as instead does the gaussian distribution for
the lowest bin.
Having calculated the σ-levels with the Pearson tem- 0.0015
plate,wecannowshowourfinalresultsintherightpanel (cid:68)
4
of Fig. 4. We have used larger bins as compared to the (cid:45)
6,
plot in the left panel because the bins extend to lower (cid:45)0.0010
(cid:72)
(cid:61)
robustness values, which have lower statistics. Also, the
R
bin height values h have been translated and scaled ac- (cid:68)
k
cording to: 0.0005
h −h¯
h −→ k k , (32)
k σ
hk 0.0000 0.00002 0.00004 0.00006 0.00008 0.0001
so as to uniformly show the signal across the various (cid:68)R(cid:61)(cid:72)(cid:45)20,(cid:45)6(cid:68)
bins. Fig.4clearlyshowshowthebodyoftheiR-PDFof
D passes the robustness test, as opposed to the low- Figure 6. Correlated distribution of bin heights for the two
EdS
robustness tail which is detected as being systematics- lowest-robustness bins of Fig. 5. The 1, 2 and 3σ confidence-
driven. More precisely, the D datum relative to the levelcontourshavebeencalculatedusingthegaussianvalues
EdS
∆χ2 =2.30,6.17,11.8. ThebinheightcorrespondingtoD
bin(−20,−6]liesat4.2-σconfidencelevels,whiletheda- EdS
is shown as an orange disk. See Section IVA3 for details.
tum relative to the robustness bin (−6,−4] lies at 2.8-σ
confidence levels. Note that in the previous results we
did not include the error in the datum relative to D .
EdS
The latter is indeed negligible as the iR-PDF of D
EdS respectively,wefindthatthesignalis>3.3σ and>2.3σ
has been found with a statistics much higher than the
at 95% confidence level, respectively.
one relative to the iR-PDF of the mocks.
Finally,onewouldexpectnon-negligiblefluctuationsin
the third (skewness) and forth (kurtosis) moments com-
ingfromasampleofonly100data,andinordertoassess 3. Taking correlations into account
theuncertaintyontheexclusionsignal,onemayproceed
as follows. Repeat enough times: a) generate 100 ran- When interpreting the results in Fig. 4 one cannot ne-
dom values from the fitted Pearson PDF; b) fit again glect the correlations among the bins ∆R . In other
k
the Pearson PDF to this new data and calculate the σ words, to get a reliable estimate by eye one is supposed
confidence level. If we now apply this routine to the two to consider only one bin at a time (namely the bin with
lowestbinsofFig.5forwhichthesignalis4.2σand2.8σ, the strongest signal) and is not allowed to combine the
10
PDF0000....01125050 mag344505 (cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:224)(cid:224)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:224)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230) (cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230) (cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)(cid:230) (cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:230)(cid:224)(cid:230)
(cid:224)(cid:224)
30
(cid:224)
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4
0.00
0 2 4 6 8 10 z
(cid:45)2R
freq
Figure 8. Plotted is the distance modulus of the SNe that
passedthequalitycuts(blue)andmadeitintothefinalUnion
Figure7. BinnediR-PDFusingfortherobustnessitsfrequen-
tist limit −2R ≡χ2 −χ2−χ2. Distributions relative to 2.1 catalogue, and the distance modulus of the SNe that did
freq btot b1 b2 not (red). See Section IVB for more details.
two unbiased mock catalogues are shown in red and blue,
while the binned χ2 distribution with 3 d.o.f. is shown with
a black curve. The error bars stand for 3σ. It is clear that
the χ2 distribution with 3 d.o.f. is not the correct PDF even
though it captures the overall shape. See Section IVA4 for fortwomockcatalogues(redandblue)togetherwiththe
more details. binned χ2 distribution with 3 d.o.f. (black). Note that
whenthebestfitsofd andd coincideonehasR =0.
1 2 freq
signals from various bins as (all) the bins are most likely
B. Union2.1 dataset with previously-excluded SNe
correlated. The same procedure can also be followed nu-
merically to get the simplest conservative estimate (see
To further test our systematic search for systematic
previous Section).
biases, it would be interesting to apply the internal ro-
Asthesignalisconcentratedinthelow-robustnesstail,
bustness method to a dataset for which one indeed ex-
wherethegaussianassumptionisnotagoodapproxima-
pectsa priori asignificantamountofcontaminationdue
tion,wecomputethecorrelationbybuildingahistogram.
to systematics. Luckily one such sample is readily avail-
In Fig. 6 we show the corresponding contour plot for the
able. The Union2.1 catalogue was in fact constructed by
twolowest-robustnessbinstogetherwiththecorrespond-
enforcing group quality criteria to their full supernova
ingdatumofD (orangedisk). Aswithonly100mocks
EdS set of 753 elements, which resulted in the removal of 173
itisnotpossibletoobtainagoodenoughPDFwhichcan
SNIa. The criteria were [8]:
then be integrated to calculate the contours, in Fig. 6
the contours are drawn using gaussian values, to wit:
1. thattheCMB-centricredshiftisgreaterthan0.015;
∆χ2 = 2.30, 6.17, 11.8. From this analysis it looks as if
taking correlations into consideration would increase the 2. that there is at least one point between −15 and 6
signal. restframe days from B-band maximum light;
3. that there are at least five valid data points;
4. Frequentist limit
4. that the entire 68% confidence interval for the
SALT2 parameter x lies between −5 and +5;
1
If one neglects the logarithmic part in Eq. (30), then
the robustness for the unbiased mocks becomes the pa- 5. data from at least 2 bands with rest-frame central
rameter goodness-of-fit test [27, 28] which is distributed wavelength coverage between 2900 Å and 7000 Å;
as a χ2 distribution with 3 degrees of freedom. As ex-
plainedinSectionIIC,however, thisisexactlytrueonly 6. at least one band redder than rest-frame U-band.
for the external robustness, but not for the internal ro-
bustness. Nevertheless, the χ2 distribution with 3 d.o.f. Now, these 173 SNIa are precisely ones for which sys-
tematics could be a dominant factor. However, for 38
capturestheoverallbehaviorofafiducialiR-PDF,ascan
of these the lightcurve fitter algorithm did not converge,
be seen in Fig. 7 where is plotted the quantity
so we do not have a measurement of their distance mod-
Rfreq ≡ −21(cid:0)χb2tot−χb21−χb22(cid:1) (33) uwliit.hW13e5thsuuspearnnaolyvazeedththaetdUindionno2t.p1acsastathloegiruequaaulgitmyecnuttesd.