Table Of Content

Multiprocessor Approximate Message Passing with Column-Wise Partitioning ∗ Yanting Ma Yue M. Lu North Carolina State University Harvard University [email protected] [email protected] Dror Baron North Carolina State University [email protected] 7 1 0 Abstract 2 Solvingalarge-scaleregularizedlinearinverseproblemusingmultipleprocessorsisimportant n invariousreal-worldapplicationsdue tothelimitationsofindividualprocessorsandconstraints a J on data sharing policies. This paper focuses on the setting where the matrix is partitioned 0 column-wise. We extend the algorithmic framework and the theoretical analysis of approxi- 3 mate message passing (AMP), an iterative algorithmfor solving linear inverse problems, whose asymptotic dynamics are characterized by state evolution (SE). In particular, we show that ] T column-wisemultiprocessorAMP(C-MP-AMP)obeysanSEunderthesameassumptionswhen theSEforAMPholds. TheSEresultsimplythat(i)theSEofC-MP-AMPconvergestoastate I s. that is no worse than that of AMP and (ii) the asymptotic dynamics of C-MP-AMP and AMP c can be identical. Moreover,for a setting that is not coveredby SE, numericalresults show that [ damping can improve the convergence performance of C-MP-AMP. 2 v 8 1 Introduction 7 5 Many scientific and engineering problems can be modeled as solving a regularized linear inverse 2 0 problem of the form 1. y = Ax+w, (1) 0 7 where the goal is to estimate the unknown x RN given the matrix A Rn×N and statistical ∈ ∈ 1 information about the signal x and the noise w Rn. : ∈ v In some scenarios, it might be desirable to partition the matrix A either column-wise or row- Xi wise and store the sub-matrices at different processors. The partitioning style depends on data availability, computational considerations, and privacy concerns. For example, in high-dimensional r a settings where N n, or in situations where the columns of A, which represent features in feature ≫ selection problems [2], cannot be shared among processors for privacy preservation, column-wise partitioning might be preferable. In this paper, we consider multiprocessor computing for the (non-overlapping) column-wise partitioned linear inverse problem: P y = A x +w, (2) p p p=1 X where P is the number of processors, A Rn Np is the sub-matrix that is stored in Processor p, p × ∈ and P N = N. p=1 p Many studies on solving the column-wise partitioned linear inverse problem (2) have been in the P context of distributed feature selection. Zhouet al. [3] modeled feature selection as a parallel group ∗This document serves as a supporting document for [1] 1 testing problem. Wang et al. [4] proposed to de-correlate the data matrix before partitioning, and each processor then works independently using the de-correlated matrix without communication with other processors. Peng et al. [5] studied problem (2) in the context of optimization, where they proposed a greedy coordinate-block descent algorithm and a parallel implementation of the fast iterative shrinkage-thresholding algorithm (FISTA) [6]. Our work is based on the approximate message passing (AMP) framework [7]. AMP is an efficient iterative algorithm for solving linear inverse problems (1). In the large scale random setting, its average asymptotic dynamics are characterized by a state evolution (SE) formalism [8], which allows one to accurately predict the average estimation error at every iteration. Recently, a finite-sample analysis of AMP [9] showed that when the prior distribution of the input signal x has i.i.d. sub-Gaussian entries,1 the average performance of AMP concentrates to the SE prediction at an exponential rate in the signal dimension N. Our goal is to extend the AMP algorithmic framework and the SE analysis in [9] to the column- wise partitioned linear inverse problem (2). We show that column-wise multiprocessor AMP (C- MP-AMP) obeys a new SE under the same model assumptions where the SE for AMP holds. With the new SE, we can predict the average estimation error in each processor at every iteration. Moreover, the comparison between the SE of AMP and that of C-MP-AMP implies that (i) the estimation error of C-MP-AMP is no worse than that of AMP and (ii) with a specific communication schedule between the processors and the fusion center that coordinates the processors, the asymptotic dynamics of C-MP-AMP are identical to that of AMP. This result implies a speedup linear in the number of processors. It is worth mentioning that row-wise multiprocessor AMP [10–12] obeys the same SE as AMP, because it distributes the computation of matrix-vector multiplication among multiple processors and aggregates the results before any other operations. Some existing work on row-wise multipro- cessorAMP [12–14]introduceslossycompressiontothecommunication between processorsandthe fusion center, whereas we assume perfect communication and focus on the theoretical justifications and implications of the new SE of C-MP-AMP. The remainder of the paper is organized as follows. Section 2 introduces the C-MP-AMP algorithm(Algorithm 1), thestate evolution sequences, andourmain performanceguarantee (Theorem 1), which is a concentration result for PL loss functions acting on the outputs generated by Algo- rithm 1 concentrates to the state evolution prediction. Section 3 proves Theorem 1. The proof is mainly based on Lemmas 3 and 4. The proof of Lemma 3 is the same as in [9] using the result that we prove in Lemma 2. Section 4 proves Lemma 4. 2 Column-Wise Multiprocessor AMP and State Evolution 2.1 Review of AMP Approximate message passing (AMP) [7] is a fast iterative algorithm for solving linear inverse problems (1). Starting with an all-zero vector x0 as its initial estimate, at the tth iteration, AMP proceeds according to zt 1 N zt = y Axt+ − η ([xt 1 +A zt 1] ), (3) − n t′−1 − ∗ − i i=1 X xt+1 = η (xt +A zt), (4) t ∗ 1A random variable X is sub-Gaussian if there exist positive constants c and κ such that P(|X −EX| > ǫ) ≤ ce−κǫ2,∀ǫ>0. 2 where vectors with negative iteration indices are all-zero vectors, A denotes the transpose of a ∗ matrix A, η : R R is a Lipschitz function with weak derivative η , for any u RN, [u] denotes t → t′ ∈ i its ith entry. The function η acts coordinate-wise when applied to vectors. That is, the vector t (η (u ),η (u ),...,η (u )) is denoted by η (u). t 1 t 2 t N t Under the assumptions on the measurement matrix A, the signal x, the measurement noise w, and the denoising function η () as listed in [9, Section 1.1], the sequence of the estimates xt that t · { } is generated by AMP (3) (4) has the following property [9]. For all ǫ (0,1), there exist constants ∈ K ,κ >0 independent of n or ǫ, such that t t N 1 P φ(xt+1,x ) E φ(η (X +τtZ),X) ǫ K e κtnǫ2, (5) (cid:12)N i i − t (cid:12) ≥ ! ≤ t − (cid:12)(cid:12) Xi=1 (cid:2) (cid:3)(cid:12)(cid:12) where φ : R2 R (cid:12)(cid:12)is a pseudo-Lipschitz function of order 2 ((cid:12)(cid:12)PL(2)),2 X pX, Z is a standard → ∼ normal random variable that is independent of X, and τt is defined via the following recursion ((σ0)2 = δ 1E[X2], δ = n/N): − (τt)2 = σ2 +(σt)2, W (σt+1)2 = δ 1E η (X +τtZ) X 2 . (6) − t − h i (cid:0) (cid:1) Notice that (5) implies, by applying the Borel-Cantelli Lemma, the almost sure convergence result proved in [8]: N 1 lim φ(xt+1,x ) a=.s. E φ(η (X +τtZ),X) . (7) N N i i t →∞ i=1 X (cid:2) (cid:3) If we choose φ(x,y) = (x y)2, then (7) characterizes the mean square error (MSE) achieved − by AMP at each iteration. 2.2 Column-Wise Multiprocessor AMP In our proposed column-wise multiprocessor AMP (C-MP-AMP) algorithm, the fusion center col- lects vectors that represent the estimations of the portion of the measurement vector y contributed by the data from individual processors according to a pre-defined communication schedule. The sum of these vectors is computed in the fusion center and transmitted to all processors. Each processor performs standard AMP iterations with a new equivalent measurement vector, which is computed using the vector received from the fusion center. The pseudocode for C-MP-AMP is presented in Algorithm 1. 2.3 Performance Guarantee SimilartoAMP,thedynamicsoftheC-MP-AMPalgorithm canbecharacterized byanSEformula. Let (σ0,kˆ0)2 = δ 1E[X2], where δ = n/N , p = 1,...,P. For outer iterations 1 s sând inner p p− p p ∀ ≤ ≤ 2Recall the definition of PL(2) from [8]: a function f : Rm → R is said to be PL(2) if there is L > 0 such that |f(x)−f(y)|≤L(1+kxk+kyk)kx−yk, ∀x,y∈Rm, where k·k denotes theEuclidean norm. 3 Algorithm 1 C-MP-AMP Inputs to Processor p: y, A , kˆ (maximum number of inner iterations at each outer p s s=0,...,sˆ { } iteration). Initialization: x0p,kˆ0 = 0, zp0,kˆ0−1 = 0, rp0,kˆ0 = 0, p. ∀ for s = 1 : sˆdo (loop over outer iterations) At the fusion center: gs = Pu=1rus−1,kˆs−1 At Processor p: P xsp,0 = xsp−1,kˆs−1, rps,0 = rps−1,kˆs−1 for t = 0 :tˆ 1 do (loop over inner iterations) s − s,k s,k s,0 z = y g r r p s p p − − − xs,k+1 = η (xs(cid:16),k +A zs,k)(cid:17) p s,k p ∗p p rs,k+1 = A xs,k+1 zps,k Np η ([xs,k +A zs,k] ). p p − n i=1 s′,k p ∗p p i Output from Processor p: xPsˆ,kˆsˆ. p iterations 0 t kˆ , we define the sequences (σs,k)2 and (τs,k)2 as s p p ≤ ≤ { } { } (σs,0)2 = (σs 1,kˆs)2, (8) p p− (τs,k)2 = σ2 + (σs,0)2+(σs,k)2, (9) p W u p u=p X6 2 (σs,k+1)2 = δ 1E η (X +τs,kZ) X , (10) p p− s,k p − (cid:20) (cid:21) (cid:16) (cid:17) where Z is a standard normal random variable that is independent of X. With these definitions, we have the following performance guarantee for C-MP-AMP. Theorem 1. Under the assumptions listed in [9, Section 1.1], let P be a fixed integer, for p = 1,...,P, let n/N = δ (0, ) be a constant. Define N = P N . Then for any PL(2) function p p ∈ ∞ p=1 p φ: R2 R, we have ǫ (0,1), there exist constants K ,κ > 0 independent of n,ǫ, such that → ∀ ∈ s,kPs,k Np 1 P φ(xs,k+1,x ) E φ(η (X +τs,kZ),X) ǫ K e κs,knǫ2, p, (cid:12)N p,i p,i − s,k p (cid:12) ≥  ≤ s,k − ∀ (cid:12)(cid:12) p Xi=1 h i(cid:12)(cid:12) (cid:12) (cid:12)  where xs,k+1(cid:12)is generated by Algorithm 1, τs,k is defined in (8–(cid:12)10), X p , and Z is a standard p (cid:12) p (cid:12) ∼ X normal random variable that is independent of X. Remark 1: C-MP-AMP converges to a fixed point that is no worse than that of AMP. This statement can be demonstrated as follows. When C-MP-AMP converges, the quantities in (8–10) do not keep changing, hence we can drop all the iteration indices for fixed point analysis. Notice that the last term on the right hand side (RHS) of (9) vanishes, which leaves the RHS independent of p. That is, (τs,k)2 are equal for all p, hence we can further drop the processor index for (τs,k)2. p p Denote (τs,k)2 by τ2 for all s,k,p, and plug (10) into (9), then p P τ2 = σ2 + δ 1E (η(X +τZ) X)2 W p− − Xp=1 h i (=a) σ2 +δ 1E (η(X +τZ) X)2 , W − − h i 4 whichisidenticaltothefixedpointequationobtainedfrom(6). Intheabove,step(a)holdsbecause P δ 1 = P Np = N. Because AMP always converges to the worst fixed point of the fixed p=1 p− p=1 n n point equation (6) [15], the average asymptotic performance of C-MP-AMP is identical to AMP P P when there is only one solution to the fixed point equation, and at least as good as AMP in case of multiple fixed points. Remark 2: The asymptotic dynamics of C-MP-AMP can be identical to AMP with a specific communication schedule. This can be achieved by letting kˆ = 1, s. In this case, the quantity s s,k ∀ (τ ) is involved only for t = 0. Because the last term in (9) is 0 when t = 0, the computation of p (τs,0)2 is independent of p. Therefore, τs,0 are again equal for all p. Dropping the processor index p p for (τs,k)2, the recursion in (8–10) can be simplified as p P (τs,0)2 = σ2 + δ 1E η (X +τs,0Z) X 2 W p− s,0 − Xp=1 h(cid:0) (cid:1) i = σ2 +δ 1E η (X +τs 1,0Z) X 2 , W − s−1,0 − − h i (cid:0) (cid:1) where the iteration evolves over s, which is identical to (6) evolving over t. Remark 3: Theorem 1 implies almost sure convergence. Similar to the performance guarantee for AMP [9], the concentration result in Theorem 1 implies Np 1 lim φ(xs,k+1,x ) a=.s.E φ(η (X +τs,kZ),X) , p, N Np p,i p,i s,k p ∀ →∞ Xi=1 h i by the Borel-Cantelli Lemma. 3 Proofs of Theorem 1 Our proof follows closely from the proof for AMP in [9], with additional dependence structure to be addressed due to vectors being transmitted among processors. 3.1 Proof Notations Withoutloss of generality, weassumethesequence kˆ inAlgorithm 1tobeaconstant valuekˆ. s s 0 Let t = skˆ+k, θ(t)= t/kˆ kˆ. Given w Rn, x {RN}p,≥for p = 1,...,P, define the column vectors p ht+1,qt RNp and bt,⌊mt ⌋ Rn for t ∈ 0 recur∈sively as follows. Starting with initial condition p p ∈ p p ∈ ≥ q0 RNp: p ∈ ht+1 = A mt qt, qt = f (ht,x ) p ∗p p− p p t p p bt = A qt λtmt 1, mt = bt + bθ(t) w (11) p p p− p p− p p u − u=p X6 where Np 1 f (ht,x ) = η (x ht) x , and λt := f (ht ,x ). (12) t p t−1 p− − p p δpNp t′ p,i p,i i=1 X In (12), the derivative of f : R2 R is with respect to the first argument. We assume that η is t t → Lipschitz for all t 0, then it follows that f is Lipschitz for all t 0. Consequently, the weak t ≥ ≥ derivativeandf exit. Further,f isassumedtobedifferentiable, exceptpossiblyatafinitenumber t′ t′ 5 of points, with bounded derivative whenever it exits. In (11), quantities with negative indices or with index θ(t)= 0 (i.e., t < kˆ) are defined to be zeros. To see the equivalence between Algorithm 1 and the recursion defined in (11) and (12), we let x0 = 0, r0 = 0, zt = 0, and p p p ht+1 = x (A zt +xt), qt = xt x , p p− ∗ p p p p− p bt = rt A x , mt = zt. p p− p p p − p Let (σ0)2 = δ 1E[X2]. We assume that (σ0)2 is strictly positive for all p = 1,...,P and for all p p− p ǫ (0,1), there exist K,κ > 0 such that ∈ q0 2 P k pk (σ0)2 ǫ Ke κnǫ2, p = 1,...,P. (13) (cid:12) n − p (cid:12)≥ ! ≤ − ∀ (cid:12) (cid:12) (cid:12) (cid:12) Define the state evolution(cid:12)scalars τt (cid:12) and σt for the the recursion defined in (11) as (cid:12) { p}t≥(cid:12)0 { p}t≥1 follows: 1 (τt)2 = (σt)2+ (σθ(t))2+σ2 , (σt)2 = E f (τt 1Z,X) 2 , (14) p p u W p δ t p− p Xu6=q h(cid:0) (cid:1) i where Z (0,1) and X p are independent. Notice that with the equivalence between X ∼ N ∼ Algorithm 1 and the recursion 11, the state evolution scalars defined in (14) matches (8) - (10). Writing the updating equations for bt,ht+1 defined in (11) in matrix form, we have p p Xt = A Mt, Yt = A Qt, (15) p ∗p p p p p where Xt =[h1 +q0 h2 +q1 ht +qt 1], Yt = [b0 b1+λ1m0 bt 1+λt 1mt 2] p p p| p p|···| p p− p p| p p p|···| p− p− p− Mt =[m0 m1 mt 1], Qt = [q0 q1 qt 1]. p p| p|···| p− p p| p|···| p− Let (mt) and (qt) denote the projection of mt and qt onto the column space of Mt and Qt, p p p p p p || || respectively. That is, (mtp) = Mpt (Mpt)∗Mpt −1(Mpt)∗mtp || (qpt) = Qtp (cid:0)(Qtp)∗Qtp −(cid:1)1(Qtp)∗qpt. || Let (cid:0) (cid:1) αt = (αt ,αt ,...,αt ) , γt = (γt ,γt ,...,γt ) (16) p p,0 p,1 p,t 1 ∗ p p,0 p,1 p,t 1 ∗ − − be the coefficient vectors of these projections. That is, αtp = (Mpt)∗Mpt −1(Mpt)∗mtp, γpt = (Qtp)∗Qtp −1(Qtp)∗qpt. (17) and (cid:0) (cid:1) (cid:0) (cid:1) t 1 t 1 − − (mt) = αt mi, (qt) = γt qi. (18) p p,i p p p,i p || || i=0 i=0 X X Define (mt) = mt (mt) , (qt) = qt (qt) . (19) p ⊥ p− p || p ⊥ p− p || The main lemma will show that αt and γt concentrate around some constant αˆt and γˆt, respec- p p p p tively. We define these constants in the following subsection. 6 3.2 Concentrating Constants Let Z˜t and Z˘t each bea sequence of zero-mean jointly Gaussian random variables whose { p}t≥0 { p}t≥0 covariance is defined recursively as follows. For t,r 0, ≥ E˜r,t E˘r,t E[Z˘rZ˘t]= p , E[Z˜rZ˜t] = p , (20) p p σrσt p p τrτt p p p p where E˘r,t = E˜r,t+ E˜θ(r),θ(t) +σ2 , p p u W u=p X6 E˜r,t = δ 1E f (τr 1Z˜r 1,X)f (τt 1Z˜t 1,X) . (21) p p− r p− p− t p− p− h i Moreover, Z˜r is independent of Z˜t and Z˘r is independent of Z˘t for all r,t 0 whenever p = q. p q p q ≥ 6 Note that according to the definition of σt and τt in (14), we have E˘t,t = (τt)2, E˜t,t = (σt)2, p p p p p p and E[(Z˜t)2] = E[(Z˘t)2] = 1. In (21), quantities with negative indices or with either θ(t) = 0 or p p θ(r)= 0 are zeros. Define matrices C˜t,C˘t Rt t,p = 1,2, such that p p ∈ × [C˜t] = E˜r,s, [C˘t] = E˘r,s, r,s = 0,...,t 1. p r+1,s+1 p p r+1,s+1 p ∀ − Define vectors E˜t,E˘t Rt,p = 1,2, such that p p ∈ E˜t = (E˜0,t,E˜1,t,...,E˜t 1,t), E˘t = (E˘0,t,E˘1,t,...,E˘t 1,t). p p p p− p p p p− Define the concentrating values αˆt and γˆt as p p γˆt = (C˜t) 1E˜t, αˆt = (C˘t) 1E˘t. (22) p p − p p p − p Let (σ0)2 = (σ0)2 and (τ0)2 = (τ0)2, and for t > 0, define p p p p ⊥ ⊥ (σt)2 = (σt)2 (γˆt) E˜t = (σt)2 (E˜t) (C˜t) 1E˜t, p ⊥ p − p ∗ p p − p ∗ p − p (τt)2 = (τt)2 (αˆt) E˘t = (σt)2 (E˘t) (C˘t) 1E˘t. (23) p ⊥ p − p ∗ p p − p ∗ p − p Lemma 1. The matrices C˜t and C˘t, t 0, defined above are invertible, and the scalars (σt)2 p p ∀ ≥ p and (τt)2, t 0, defined above are strictly positive. ⊥ p ∀ ≥ ⊥ Proof. Theprooffor C˜t beinginvertible and(σt)2 beingstrictly positive is thesame as in [9]. Now p p consider C˘t+1. Notice that C˘t+1 is the sum of ⊥a positive definite matrix (C˜t+1) and P positive p p p semi-definite matrices, hence, C˘t+1 is positive definite. Consequently, p det(C˘t+1)= det(C˘t)det((τt)2 (E˘t) (C˘t) 1E˘t) > 0, (24) p p p − p ∗ p − p which implies (τt)2 (E˘t) (C˘t) 1E˘t = (τt)2 > 0. p − p ∗ p − p p ⊥ 7 3.3 Condition Distribution Lemma Let the sigma algebra St1,t be generated by x,w,b0,...,bt1 1,m0,...,mt1 1,h1,...,ht,q0,...,qt, p. p p − p p − p p p p ∀ We now compute the conditional distribution of A given St1,t for 1 p P, where t is either t p 1 ≤ ≤ or t+1. Notice that conditioning on St1,t is equivalent to conditioning on the linear constraints: A Qt1 = Yt1, A Mt = Xt, 1 p P, (25) p p p ∗p p p ≤ ≤ where in (25), only A , 1 p P, are treated as random. p ≤ ≤ Let PkQtp1 = Qtp1((Qtp1)∗Qtp1)−1Qtp1 and PkMpt = Mpt((Mpt)∗Mpt)−1Mpt, which are the projectors onto the column space of Qt1 and Mt, respectively. The following lemma provides the conditional p p distribution of the matrices A , p = 1,...,P, given t1,t. p G Lemma 2. For t = t or t+1, the conditional distribution of the random matrices A , p = 1,...,P, 1 p given St1,t satisfies (A ,...,A ) =d (Et1,t+P A˜ P ,...,Et1,t+P A˜ P ) 1 P |St1,t 1 ⊥M1t 1 ⊥Qt11 P ⊥MPt P ⊥QtP1 where P⊥Qtp1 = I−PkQtp1 and P⊥Mpt = I−PkMpt. A˜p =d Ap and A˜p is independent of St1,t. Moreover, A˜ is independent of A˜ for p = q. Et1,t is defined as p q p 6 Et1,t = Yt1((Qt1) Qt1) 1(Qt1) +Mt((Mt) Mt) 1(Xt) p p p ∗ p − p ∗ p p ∗ p − p ∗ Mt((Mt) Mt) 1(Mt) Yt1((Qt1) Qt1) 1(Qt1) . − p p ∗ p − p ∗ p p ∗ p − p ∗ Proof. To simplify the notation, we drop the superscriptt or t in the following proof. It should be 1 understood that Q represents Qt1, Y represents Yt1, M represents Mt, and X represents Xt. p p p p p p p p First let us consider projections of a deterministic matrix. Let Aˆ be a deterministic matrix p that satisfies the linear constraints Y = Aˆ Q and X = Aˆ M , then we have p p p p ∗p p Aˆ = Aˆ Q (Q Q ) 1Q +Aˆ (I Q (Q Q ) 1Q ), p p p ∗p p − ∗p p − p ∗p p − ∗p Aˆ = M (M M ) 1M Aˆ +(I M (M M ) 1M )Aˆ . p p p∗ p − p∗ p − p p∗ p − p∗ p Combining the two equations above, as well as the two linear constraints, we can write Aˆ = Y (Q Q ) 1Q +M (M M ) 1X M (M M ) 1M Y (Q Q ) 1Q +P Aˆ P . (26) p p ∗p p − ∗p p p∗ p − p − p p∗ p − p∗ p ∗p p − ∗p ⊥Mp p ⊥Qp We now demonstrate the conditional distribution of A ,...,A . Let S ,...,S bearbitrary Borel 1 P 1 P sets on Rn N1,...,Rn NP, respectively. × × P A S ,...,A S A Q = Y ,A M = X , p 1 ∈ 1 P ∈ P p p p ∗p p p ∀ (=a(cid:0)) P Et1,t+P A P (cid:12) S ,...,Et1,t+P A P (cid:1) S A Q = Y ,A M = X , p 1 ⊥M1 1 ⊥Q1(cid:12)∈ 1 P ⊥MP P ⊥QP ∈ P p p p ∗p p p ∀ (=b) P (cid:16)Et11,t+P⊥M1A1P⊥Q1 ∈ S1,...,EtP1,t+P⊥MPAPP⊥QP ∈ SP (cid:12)(cid:12) (cid:17) (cid:16) (cid:17) = P Et1,t+P A P S ...P Et1,t+P A P S , (27) 1 ⊥M1 1 ⊥Q1 ∈ 1 P ⊥MP P ⊥QP ∈ P (cid:16) (cid:17) (cid:16) (cid:17) which implies the desired result. In step (a), Et1,t = Y (Q Q ) 1Q +M (M M ) 1X M (M M ) 1M Y (Q Q ) 1Q , p = 1,...,P, p p ∗p p − ∗p p p∗ p − p∗ − p p∗ p − p∗ p ∗p p − ∗p 8 which follows from (26). Step (b) holds since P A P is independent of the conditioning. The ⊥Mp p ⊥Qp independence is demonstrated as follows. Notice that ApQp = ApP|Q|pQp. In what follows, we will show that A|p| := ApP|Q|p is independent of A⊥r := ArP⊥Qr, for p,r = 1,...,P. Then similar approach can be used to demonstrate that P⊥MpAp is independent of P|M| rAr. Together they provide the justification for step (b). Note that A|p| and A⊥r are jointly normal, hence it is enough to show they are uncorrelated. N N E [A|p|]i,j[A⊥r ]m,l = E [Ap]i,k[PkQp]k,j [Ar]m,k Ik,l−[PkQr]k,l ( ! !) n o Xk=1 Xk=1 (cid:16) (cid:17) N N (=a) n1δ0(i,m)δ0(p,r) [PkQp]k,jIk,l− [PkQp]k,j[PkQtr1]k,l! k=1 k=1 X X N (=b) n1δ0(i,m)δ0(p,r) [PkQp]l,j − [PkQp]k,j[PkQr]l,k (=c) 0, ! k=1 X where δ (i,j) is the Kronecker delta function. In the above, step (a) holds since the original matrix 0 A has (0,1/n) i.i.d. entries, step (b) holds since projectors are symmetric matrices, and step (c) N follows the property of projectors P2 = P. Combining the results in Lemma 2 and [9, Lemma 4], we have the following conditional distribution lemma. Lemma 3. For the vectors ht+1 and bt defined in (11), the following holds for t 1, p = 1,...,P: p p ≥ b0 S0,0 =d (σ0) Z′0+∆0,0, h1 S1,0 =d (τ0) Z0+∆1,0, (28) p| p ⊥ p p p| p ⊥ p p t 1 t 1 bt St,t =d − γˆt bi +(σt) Z′t+∆t,t, ht St+1,t =d − αˆt hi+1+(τt) Zt +∆t+1,t, (29) p| p,i p p ⊥ p p p| p,i p p ⊥ p p i=0 i=0 X X where (q0) ∆0,0 = k p ⊥k (σ0) Z′0 (30) p √n − p ⊥! p 1 (m0) (m0) q0 2 − (b0) (m0) q0 2 ∆1p,0 = " k √pn⊥k −(τp0)⊥!I− k √pn⊥kPkqp0#Zp0+qp0 k npk ! p ∗n p ⊥ − k npk ! (31) ∆tp,t = t−1 γpt,i−γˆpt,i bip+" k(q√ptn)⊥k −(σpt)⊥!I− k(q√pt)n⊥kPkMpt#Zp′t i=0 X(cid:0) (cid:1) +Mpt (Mptn)∗Mpt −1 (Hpt)∗n(qpt)⊥ − (Mnpt)∗ λtpmtp−1− t−1λtp,iγpt,imip−1 (32) ! " #! i=1 X ∆tp+1,t = t−1 αtp,i−αˆtp,i hip+1+" k(m√tpn)⊥k −(τpt)⊥!I− k(m√tpn)⊥kPkQtp+1#Zpt i=0 X(cid:0) (cid:1) +Qt+1 (Qtp+1)∗Qtp+1 −1 (Bpt+1)∗(mtp)⊥ (Qtp+1)∗ qt t−1αt qi , (33) p n n − n p− p,i p ! " #! i=0 X 9 where Z′t Rn and Zt RNp are random vectors with independent standard normal elements, and p ∈ p ∈ are independent of the corresponding sigma algebras. Moreover, Z′t is independent of Z′t and Zt p q p is independent of Zt when p =q. q 6 Proof. The proof for each individual p [P] is similar to the proof for [9, Lemma 4]. The claim that Z′t is independent of Z′t and Zt ∈is independent of Zt when p = q follows from Lemma 2, p q p q 6 where we have that A˜ is independent of A˜ for p = q. p q 6 3.4 Main Concentration Lemma . We use the shorthand X = c to denote the concentration inequality P (X c ǫ) K e κtnǫ. n n t − | − | ≥ ≤ As specified in the theorem statement, the lemma holds for all ǫ (0,1), with K ,κ denoting the t t ∈ generic constants dependent on t, but not on n,ǫ. . Lemma 4. With the = notation defined above, the following holds for all t 0, p = 1,...,P. ≥ (a) ∆t,t 2 P k p k ǫ Kte−κtnǫ. (34) n ≥ ≤ ! ∆t+1,t 2 P k p k ǫ Kte−κtnǫ. (35) n ≥ ≤ ! (b) (i) For pseudo Lipschitz functions φ :Rt+2 R, h → Np 1 . φ (h1 ,...,ht+1,x )= E[φ (τ0Z˜0,...,τtZ˜t,X)]. (36) N h p,i p,i p,i h p p p p p i=1 X (ii) Let ψ : R2 R be a bounded function that is differentiable in the first argument except h → possibly at a finite number of points, with bounded derivative when it exists. Then, Np 1 . ψ (ht+1,x ) = E[ψ (τtZ˜t,X)], (37) N h p,i p,i h p p p i=1 X where Z˜t is defined in (20), and X p is independent of Z˜ t. { p} ∼ X { }p (iii) For pseudo-Lipschitz function phi :RP(t+1)+1 R, b → n 1 . φ (b0 ,...,b0 ,...,bt ,...,bt ,w )= E φ (σ0Z˘0,...,σ0Z˘0,...,σtZ˘t,...,σt Z˘t ,W) , n b 1,i P,i 1,i P,i i b 1 1 P P 1 1 P P Xi=1 h i (38) where Z˘t is defined in (20), and W p is independent of Z˘t . { p} ∼ W { p} (c) (htp+1)∗qp0 . (htp+1)∗xp . = 0, = 0. (39) n n (btp)∗w . = 0. (40) n 10