Random Subsets of Structured Deterministic Frames have MANOVA Spectra

0.55 MB·

by  Marina Haikin

Random Subsets of Structured Deterministic
Frames have MANOVA Spectra
Marina Haikin Ram Zamir Matan Gavish
∗ ∗ †
7
1
0
Abstract
2

n We draw a random subset of k rows from a frame with n rows (vectors)
a
and m columns (dimensions), where k and m are proportional to n. For a
J
5 varietyofimportantdeterministicequiangulartightframes(ETFs)andtight
non-ETF frames, we consider the distribution of singular values of the k-

]
T subset matrix. We observe that for large n they can be precisely described
I by a known probability distribution – Wachter’s MANOVA spectral distri-
.
s bution,aphenomenonthatwaspreviouslyknownonlyfortwotypesofran-
c
[ domframes. In terms ofconvergence to this limit, the k-subsetmatrix from

all these frames is shown to behave exactly like the classical MANOVA (Ja-
1
v cobi) random matrix ensemble. Thus empirically the MANOVA ensemble
1 offers a universal description of spectra of randomly selected k-subframes,
1
eventhosetakenfromdeterministicframes. Thesameuniversalityphenom-
2
1 enaisshowntoholdfornotablerandomframesaswell. Thisdescriptionen-
0 ablesexactcalculations ofpropertiesofsolutionsforsystemsoflinearequa-
.
1 tionsbasedonarandomchoiceofk frame vectorsoutofnpossiblevectors,
0
and has a variety of implications for erasure coding, compressed sensing,
7
1 andsparserecovery. Our resultsareempirical, buttheyare exhaustive,pre-
: ciseandfullyreproducible.
v
i
X
Keywords. Deterministicframes—equiangulartightframes—MANOVA—Jacobien-
r
a semble—restrictedisometryproperty—Gaussianchannelwitherasures—Grassman-
nian frame — Paley frame — random Fourier — Shannon transform — analog source
coding.
Consider a frame x n Rm or Cm and stack the vectors as rows to obtain
the n-by-m frame {mai}tri=ix1 X⊂. Assume that x = 1 (deterministic frames) or
lim x = 1 almost surely (random fra|m| ei|s|)2. This paper studies properties
n i
of a→r∞ankdomk subframe x , where K is chosen uniformly at random from
i i K
{ }∈
[n] = 1,...,n and K = k n. We let X denote the k-by-m submatrix of X
K
{ } | | ≤
created by picking only the rows x ; call this object a typical k-submatrix of
i i K
{ }∈
X. We consider a collection of well-known deterministic frames, listed in Table
∗EE-SystemsDepartment,TelAvivUniversity,TelAviv,Israel
†SchoolofComputerScienceandEngineering,HebrewUniversityofJerusalem
1
2
1, which we denote by . Most of the frames in are equiangular tight frames
X X
(ETFs), and some are near-ETFs.
This paper suggests that for a frame in it is possible to calculate quanti-
X
ties of the form E Ψ(λ(G )), where λ(G ) = (λ (G ),...,λ (G )) is the vector
K K K 1 K k K
of eigenvalues of the k-by-k Gram matrix G = X X and Ψ is a functional of
K K K′
these eigenvalues. As discussed below, such quantities are of considerable inter-
est in various applications where frames are used, across a variety of domains,
includingcompressed sensing, sparse recovery anderasure coding.
We present a simple and explicit formula for calculating E Ψ(λ(G )) for a
K K
given frame in and a given spectral functional Ψ. Specifically,
X
E Ψ(λ(G )) Ψ fMANOVA ,
K K ≈ β,γ
where β = k/m, γ = m/n and where fMAN(cid:0)OVA is the d(cid:1)ensity of Wachter’s classi-
β,γ
calMANOVA(β,γ)limitingdistribution[27]. Thefluctuationsaboutthisapprox-
imate valueare given exactlyby
E Ψ(λ(G )) Ψ fMANOVA 2 = Cn blog a(n). (1)
K K − β,γ − −
(cid:12) (cid:0) (cid:1)(cid:12)
While the consta(cid:12)nt C may depend on the fr(cid:12)ame, the exponents a and b are uni-
versal and depend only on Ψ and on the aspect ratios β and γ. Evidently, the
precision of the MANOVA-based approximation is good, known, and improves
asmandk both grow proportionally to n.
Formula 1 is based on a far-reaching universality hypothesis: For all frames in
, as well as for well-known random frames also listed in Table 1, we find that
X
the spectrum of the typical k-submatrix ensemble is indistinguishable from that
of the classical MANOVA (Jacobi) random matrix ensemble [1] of the same size.
(Interestingly, it will be shown that for deterministic ETFs this indistinguishably
holdsina stronger sensethan for deterministic non-ETFframes.) Thisuniversal-
ity isnot asymptotic, and concerns finite n-by-m frames. However, it does imply
that the spectrum of the typical k-submatrix ensemble converges to a universal
limitingdistribution, whichisnonotherthanWachter’sMANOVA(β,γ)limiting
distribution[27]. Italsoimpliesthattheuniversalexponentsaandbin(1)arepre-
viously unknown, universal quantities corresponding to the classical MANOVA
(Jacobi) random matrix ensemble.
Here, we test Formula 1 and the underlying universality hypothesis by con-
ducting substantial computer experiments, in which a large number of random
k-submatrices are generated. We study a large variety of deterministic frames,
bothrealandcomplex. Inadditiontotheuniversalobject(theMANOVAensem-
ble) itself, we study difference-set spectrum frames, Grassmannian frames, real
Paley frames, complex Paley frames, quadratic phase chirp frames, Spikes and
Sinesframes, andSpikes andHadamard frames.
Wereportcompellingempiricalevidence,systematicallydocumentedandan-
alyzed, which fully supports the universality hypothesis and (1). Our results are
empirical, but they are exhaustive, precise, reproducible and meet the best stan-
dardsofempirical science.
3
Forthispurpose,wedevelopanaturalframeworkforempiricallytestingsuch
hypothesesregardinglimitingdistributionandconvergenceratesofrandomma-
trix ensembles. Before turning to deterministic frames, we validate our frame-
work on well-known random frames, including real orthogonal Haar frames,
complex unitary Haar frames, real random Cosine frames and complex random
Fourier frames. Interestingly, rigorous proofs that identify the MANOVA distri-
bution as the limiting spectral distribution of typical k-submatrices can be found
in the literature for two of these random frames, namely the random Fourier
frame [6]andthe unitary Haarframe [23].
Motivation
Frames can be viewed as an analog counterpart for digital coding. They pro-
vide overcomplete representation of signals, adding redundancy and increasing
immunity to noise. Indeed, they are used in many branches of science and engi-
neeringfor stable signal representation, aswell aserror and erasure correction.
Letλ(G)denotethevectorofnonzeroeigenvaluesofG = X X andletλ (G)
′ max
and λ (G) denote its max and min, respectively. Frames were traditionally de-
min
signed to achieve frame bounds λ (G) as high as possible (resp. λ (G) as
min max
low as possible). Alternatively, they were designed to minimize mutual coher-
ence[15,31], the maximal pairwise correlation between anytwo frame vectors.
In the passing decade it has become apparent that neither frame bounds (a
globalcriterion)norcoherence(alocal,pairwisecriterion)aresufficienttoexplain
variousphenomenarelatedtoovercompleterepresentations,andthatoneshould
also look at collective behavior of k frame vectors from the frame, 2 k n.
≤ ≤
While different applications focus on different properties of the submatrix G ,
K
mostofthesepropertiescanbeexpressedasafunctionofλ(G ),andevenjustan
K
average ofa scalarfunction of the eigenvalues. Here are afewnotable examples.
Restricted Isometry Property (RIP). Recovery of any k/2-sparse signal v Rn
from its linearmeasurementF v usingℓ minimization isguaranteedifthe s∈pec-
′ 1
tral radiusof G I, namely,
K
−
Ψ (λ(G )) = max λ (G ) 1, 1 λ (G ) , (2)
RIP K max K min K
{ − − }
isuniformly bounded bysome δ < 0.4531on all K [n][2,20,26].
⊂
Statistical RIP. Numerous authors have studied a relaxation of the RIP condi-
tion suggested in [10]. Define
1 Ψ (λ(G )) δ
RIP K
Ψ (λ(G )) = ≤ . (3)
StRIP,δ K
(0 otherwise
Then E Ψ (λ(G )) is the probability that the RIP condition with bound δ
K StRIP,δ K
holdswhen X acts on a signal supported on a random set ofk coordinates.
4
Analog coding of a source with erasures. In [3] two of us considered a typical
erasure pattern of n k random samples known at the transmitter, but not the
−
receiver. The rate-distortion function of the coding scheme suggested in [3] is
determinedby E log(βΨ (λ(G ))), with
K AC K
1
Ψ (λ(G )) = tr[(G ) 1]. (4)
AC K K −
k
The quantity Ψ (λ(G )) is the signal amplification responsible for the excess
AC K
rate ofthe suggested codingscheme.
Shannontransform. The quantity
1
Ψ (λ(G )) = log(det(I +αG ))
Shannon K K
k
(5)
1
= tr(log(I +αG )),
K
k
which was suggested in [4], measures the capacity of a linear-Gaussian erasure
channel. Specifically, it assumes y = XX x + z, (where x and y are the channel
′
inputandoutput)followedbyn k randomerasures. Thequantityαin(5)isthe
−
signal-to-noise ratioSNR = α 0.
≥
In this paper, we focus on typical-case performance criteria (those that seek to
optimize E Ψ(λ(G )) over random choice of K) rather than worse-case perfor-
K K
mance criteria (those that seek to optimize max Ψ(λ(G )), such as RIP). For
K [n] K
⊂
the remainder of this paper,K [n] will denote a uniformly distributed random
⊂
subsetof size k. Importantly, k should be allowedto be large, even aslarge asm.
For a given Ψ, one would like to design frames that optimize E Ψ(λ(G )).
K K
This turns out to be a difficult task; in fact, it is not even known how to calculate
E Ψ(λ(G )) for a given frame X. Indeed, to calculate this quantity one effec-
K K
tively has to average Ψ over the spectrum λ(G ) for all n subsets K [n]. It is
K k ⊂
oflittlesurprise totheinformation theoristthatthefirstframedesigns,forwhich
(cid:0) (cid:1)
performance was formally bounded (and still not calculated exactly), consisted
ofrandom vectors [14,19–22,24].
Random Frames
When the frame is random, namely when X is drawn from some ensemble of
random matrices, the typical k-submatrix X is also a random matrix. Given a
K
specific Ψ, rather than seeking to bound E Ψ(λ(G )) for specific n and m, it can
K K
beextremely rewarding to study the limit ofΨ(λ(G )) asthe frame size nand m
K
grow. This is because tools from random matrix theory become available, which
allowexactasymptoticcalculationofλ(G )andΨ(λ(G )),andalsobecausetheir
K K
limiting values are usually very close to their corresponding values for finite n
andm, even for lowvaluesof n.
5
Let us consider then a sequence of dimensions m with m /n = γ γ and
n n n
→
a sequence of random frame matrices X(n) Rn mn or Cn mn. To characterize
× ×
⊂
the collective behavior of k-submatrices we choose a sequence k with k /m =
n n n
β β and look at the spectrum λ(G ) of the random matrix X as n ,
n → Kn Kn → ∞
where K [n] isa randomlychosen subset with K = k .
n n n
⊂ | |
Amainstayofrandom matrixtheory isthecelebrated convergence oftheem-
pirical spectral distribution of random matrices, drawn from a certain ensemble,
to a limiting spectral distribution corresponding to that ensemble. This has in-
deedbeenestablished for three random frames:
(n)
1. Gaussian i.i.d frame: Let X have i.i.d normal entries with mean zero
normal
and variance 1 . The empirical distribution of λ(G ) famously converges,
mn K
almost surely in distribution, to the Marc˘enko-Pastur density [13] with pa-
rameter β. supported on [λMP,λMP] where λMP = (1 √β)2. Moreover,
(n) − + (±n) ±
almost surely λ (G ) λ and λ (G ) λ ; in other words,
max normal → + min normal → −
the maximal and minimal empirical eigenvalues converge almost surely to
the edgesof the support of the limitingspectral distribution [28].
2. Random Fourier frame: Consider the random Fourier frame, in which the m
n
(n)
columns of X are drawn uniformly at random from the columns of
fourier
then-by-ndiscreteFouriertransform (DFT)matrix(normalizeds.tabsolute
valueofmatrixentriesis 1 ). Farrell[6]hasprovedthattheempiricaldis-
√mn
tribution of λ(G ) converges, almost surely in distribution, as n and
K
→ ∞
as m and k grow proportionally to n, to the so-called MANOVA limiting
n n
distribution, which we now describe briefly.
The classical MANOVA(n,m,k, ) ensemble1, with R,C is the dis-
F F ∈ { }
tribution of the random matrix
n
1 1
(AA′ +BB′)−2BB′(AA′ +BB′)−2 , (6)
m
whereA ,B arerandomstandardGaussiani.i.dmatriceswithen-
k (n m) k m
× − ×
tries in . Wachter [27] discovered that, as k /m β 1 and m /n γ,
n n n
F → ≤ →
the empirical spectral distribution of the MANOVA(n,m ,k ,R) ensemble
n n
converges, almost surely in distribution, to the so-called MANOVA(β,γ)
limitingspectral distribution2, whose density isgiven by
(x r )(r x)
fβM,γANOVA(x) = 2β−πx(−1 +γx−) ·I(r−,r+)(x) (7)
p −
+
1 1 1
+ 1+ δ x
β − βγ · − γ
(cid:18) (cid:19) (cid:18) (cid:19)
1Also known as the beta-Jacobi ensemble with beta=1 (orthogonal) for = R, and beta=2
(unitary)for =C. F
F
2TheliteratureusesthetermMANOVAtoreferbothtotherandommatrixensemble,whichwe
denoteherebyMANOVA(n,m,k, ),andtothelimitingspectraldistribution,whichwedenote
F
herebyMANOVA(β,γ).
6
where (x)+ = max(0,x). The limiting MANOVA distribution is compactly
supported on [r ,r ]with
+
−
2
r = β(1 γ) 1 βγ . (8)
± − ± −
(cid:18) (cid:19)
p p
Thesame holdsfor the MANOVA(n,m ,k ,C)ensemble.
n n
Note that the support of the MANOVA(β,γ) distribution is smaller than
that of the corresponding Marc˘enko-Pastur law for the same aspect ratios.
Figure 1shows these two densitiesfor β = 0.8and γ = 0.5.
1.8 1.8
1.6 1.6
1.4 1.4
1.2 1.2
1 1
0.8 0.8
0.6 0.6
0.4 0.4
0.2 0.2
0 0
0 1 2 3 4 0 0.1
Figure 1: Limiting MANOVA (β = 0.8,γ = 0.5) and Marc˘enko-Pastur (β = 0.8)
density functions. Left: density on the interval x [0,4]. Right: Zoom in on the
∈
interval x [0,0.1].
∈
(n)
3. Unitary Haar frame: Let X consist of the first m columns of a Haar-
haar n
distributed n-by-n unitary matrix normalized by n (the Haar distribu-
mn
tion beingthe uniform distribution over the group of n-by-n unitary matri-
p
ces). Edelman and Sutton [23] proved that the empirical spectral distribu-
tionofλ(G )alsoconverges,almostsurelyindistribution,totheMANOVA
K
limitingspectral distribution (Seealso [27]andthe closing remarksof [6].)
ThemaximalandminimaleigenvaluesofamatrixfromtheMANOVA(n,m,k, )
F
ensemble( R,C )areknowntoconvergealmostsurelytor andr ,respec-
+
F ∈ { } −
tively[11]. Whileweare notawareofanyparallelresults fortherandom Fourier
and Haar frames, the empirical evidence in this paper show that it must be the
case.
7
These random matrix phenomena have practical significance for evaluations
offunctions oftheform Ψ(λ(G ))such asthose mentionedabove. Thefunctions
K
Ψ and Ψ , for example, are what [29] call linear spectral statistics, namely
AC Shannon
functions of λ(G ) that maybe written as an integral of a scalarfunction against
K
the empirical measure of λ(G ). Convergence of the empirical distribution of
K
λ(G(n))to the limiting MANOVAdistribution with density fMANOVA implies
Kn β,γ
1
lim Ψ (λ(G(n))) = fMANOVA(x)dx (9)
n AC Kn x β,γ
→∞ Z
lim Ψ (λ(G(n))) = log(1+αx)fMANOVA(x)dx
n Shannon Kn β,γ
→∞ Z
forboththerandomFourierandHaarframes;theintegralsontherighthandside
maybeevaluated explicitly. Similarly, convergence ofλ (G )andλ (G )to
max K min K
r andr implies, forexample, that
+
−
(n)
lim Ψ (λ(G )) = max(r 1,1 r ). (10)
n RIP Kn + − − −
→∞
To demonstrate why such calculations are significant, we note that Equations
(9) and (10) immediately allow us to compare the Gaussian i.i.d frame with the
random Fourier and Haar frames, in terms of their limiting value of functions of
interest. Figure 2 compares the limiting value of Ψ , Ψ and Ψ over
RIP AC Shannon
varying values of β = lim k /m . The plots clearly demonstrate that frames
n n n
→∞
whosetypicalk-submatrixexhibitsaMANOVAspectrum,aresuperiortoframes
whose typical k-submatrix exhibits a Marc˘enko-Pastur spectrum, across the per-
formance measures.
Deterministic Frames: Universality Hypothesis
Deterministicframes,namelyframeswhosedesigninvolvesnorandomness,have
so far eluded this kind of asymptotically exact analysis. While there are results
regarding RIP [17,18] and statistical RIP [10,12,16], for example, of determinis-
tic frame designs, nothing analogous, say, to the precise comparisons of Figure 2
exists in the literature to the best of our knowledge. Specifically, no results anal-
ogous to (9)and (10)are known for deterministic frames, letalone the associated
convergence rates, ifany.
Inordertosubjectdeterministicframestoanasymptoticanalysis,weshiftour
focusfrom asingleframeX toafamilyofdeterministicframes X(n) createdby
{ }
a common construction. The frame matrix X(n) is n-by-m . Each frame family
n
determines allowable sub-sequences (n,m ); to simplify notation, we leave the
n
subsequenceimplicitandindexthe framesequencesimplybyn. Theframe fam-
ily also determines the aspect ratio limit γ = lim m /n. In what follows we
n n
→∞
also fix a sequence k with β = lim k /m , and let K [n] denote a uni-
n n n n n
→∞ ⊂
formly distributed random subset.
8
0.9
0.8
0.7
0.6
0.5
0.4
2 4 6 8 10 12 14 16 18 20
11
10
9
8
7
6
5
4
3
2
1.5 2 2.5 3 3.5 4
0.96
0.94
0.92
0.9
0.88
0.86
1.5 2 2.5 3 3.5 4
Figure 2: Comparison of limiting values of E Ψ(λ(G )) for the three functions
K K
Ψ discussed in Motivation Section between the Marc˘enko-Pastur limiting distri-
bution and the MANOVAdistribution. Left: Ψ (lower isbetter). Middle: Ψ
RIP AC
(lowerisbetter). Right: Ψ (higherisbetter).
Shannon
9
Table1: Framesunderstudy
Label Name RorC Naturalγ Tightframe Equiangular References
Deterministic
DSS Difference-setspectrum C Yes Yes [7]
GF Grassmannianframe C 1/2 Yes Yes [8,Cor.2.6b]
RealPF RealPaley’sconstr. R 1/2 Yes Yes [9]
ComplexPF ComplexPaley’sconstr. C 1/2 Yes Yes [8,Cor.2.6a]
Alltop QuadraticPhaseChirp C 1/L Yes No [5,eq.S4]
SS SpikesandSines C 1/2 Yes No [31]
SH SpikesandHadamard R 1/2 Yes No [31]
Random
HAAR UnitaryHaarframe C Yes No [6,23]
RealHAAR OrthogonalHaarframe R Yes No [23]
RandDFT RandomFouriertransform C Yes No [6]
RandDCT RandomCosinetransform R Yes No
Frames under study. The different frames that we studied are listed in Table 1,
inamannerinspiredby[5]. Inadditiontoourdeterministicframesofinterest(the
set ), thetable containsalsotwoexamplesofrandom frames(realandcomplex
X
variantfor each), for validation and convergence analysispurposes.
Functionalsunderstudy. WestudiedthefunctionalsΨ from(3),Ψ from
StRIP AC
(4), Ψ from (5). In addition, we studied the maximal and minimal eigen-
Shannon
valuesofG , and itscondition number:
K
Ψ (λ(G )) = λ (G )
max K max K
Ψ (λ(G )) = λ (G )
min K min K
Ψ (λ(G )) = λ (G )/λ (G ).
cond K max K min K
Measuring the rate of convergence. In order to quantify the rate of conver-
gence of the entire spectrum of the k-by-m matrix X , which is a k-submatrix of
K
an n-by-m frame matrix, to a limiting distribution, we let F[X ] denote the em-
K
pirical cumulative distribution function (CDF) of λ(G ), and let FMANOVA(x) =
K β,γ
x fMANOVA(x)dx denote the CDF of the MANOVA(β,γ) limiting distribution.
r− β,γ
Thequantity
R
∆ (X ;n,m,k) = F[X ] FMANOVA ,
KS K K − βn,γn KS
(cid:12)(cid:12) (cid:12)(cid:12)
where isthe Kolmogorov-Smirn(cid:12)(cid:12)ov (KS) distance bet(cid:12)w(cid:12) een CDFs, measures
||·||KS
thedistancetothehypothesisedlimit. Asabaselineweuse∆ (Y ;n,m,k),
KS n,m,k,
where Y is a matrix from the MANOVA(n,m,k, ) ensemble, wFith = R
n,m,k,
if X is real aFnd = C if complex. Figure 3 illustratesFthe KS-distance beFtween
K
F
anempirical CDFand the limiting MANOVACDF.
Similarly, in order to quantify the rate of convergence of a functional Ψ, the
quantity
∆ (X ;n,m,k) = Ψ(λ(G )) Ψ(fMANOVA)
Ψ K K − βn,γn
(cid:12) (cid:12)
(cid:12) (cid:12)
10
1
RandDFTempiricalcdf
0.9 MANOVAcdf
KS-distance
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
x
Figure 3: KS-distance ofrandom DFT subframe, β = 0.8,γ = 0.5,n = 100.
isthe distance between the measured value of Ψ on a given k-submatrix X and
K
its hypothesised limiting value. For a baseline we can use ∆ (Y ;n,m,k),
Ψ n,m,k,
with = R if X is real and = C if complex. For linear spectral fuFnctionals
K
F F
likeΨ andΨ ,whichmaybewrittenasΨ(λ(G )) = ψdF[X ]forsome
AC Shannon K K
kernel ψ, we have Ψ(fMANOVA) = ψdFMANOVA. For Ψ that depends on
β,γ β,γ RRIP
λ (G )and λ (G )we have Ψ (fMANOVA) = max r 1,1 r .
max K min K RIRP β,γ { + − − −}
Universality Hypothesis. The contributions of this paper are based on the fol-
lowing assertions on the typical k-submatrix ensemble X corresponding to a
Kn
frame familyX(n). Thisfamily maybe random ordeterministic, real or complex.
H1 Existence of a limiting spectral distribution. The empirical spectral distribu-
(n) (n)
tion of X , namely the distribution of λ(G ), converges, as n , to
Kn Kn → ∞
(n)
a compactly-supported limiting distribution; furthermore, λ (G ) and
max Kn
(n)
λ (G ) converge to the edgesofthat compact support.
min Kn
H2 Universality of the limiting spectral distribution. The limiting spectral distri-
(n)
bution of X is the MANOVA(β,γ) distribution [27] whose density is (7).
Kn
(n) (n)
Alsoλ (G ) r and λ (G ) r where r isgiven by (8).
max Kn → + min Kn → − ±
H3 Exact power-law rate of convergence for the entire spectrum. The spectrum of
(n)
X converges tothe limiting MANOVA(β,γ)distribution
Kn
2
E (∆ X(n);n,m ,k ) 0
Kn KS Kn n n ց
(cid:16) (cid:17)