MEMOIRS of the American Mathematical Society Volume 241 • Number 1143 (fourth of 4 numbers) • May 2016 Nil Bohr-Sets and Almost Automorphy of Higher Order Wen Huang Song Shao Xiangdong Ye ISSN 0065-9266 (print) ISSN 1947-6221 (online) American Mathematical Society MEMOIRS of the American Mathematical Society Volume 241 • Number 1143 (fourth of 4 numbers) • May 2016 Nil Bohr-Sets and Almost Automorphy of Higher Order Wen Huang Song Shao Xiangdong Ye ISSN 0065-9266 (print) ISSN 1947-6221 (online) American Mathematical Society Providence, Rhode Island Library of Congress Cataloging-in-Publication Data Names: Huang,Wen,1975–—Shao,Song,1976–—Ye,Xiangdong. Title: Nil Bohr-sets and almost automorphy of higher order / Wen Huang, Song Shao, Xiang- dongYe. Description: Providence,RhodeIsland: AmericanMathematicalSociety,2016. —Series: Mem- oirsoftheAmericanMathematicalSociety,ISSN0065-9266;volume241,number1143—Includes bibliographicalreferencesandindex. Identifiers: LCCN2015050793—ISBN9781470418724(alk. paper) Subjects: LCSH:Automorphicfunctions. —Fourieranalysis. Classification: LCC QA353.A9 H83 2016 — DDC 515/.9–dc23 LC record available at http:// lccn.loc.gov/2015050793 DOI:http://dx.doi.org/10.1090/memo/1143 Memoirs of the American Mathematical Society Thisjournalisdevotedentirelytoresearchinpureandappliedmathematics. Subscription information. Beginning with the January 2010 issue, Memoirs is accessible from www.ams.org/journals. The 2016 subscription begins with volume 239 and consists of six mailings, each containing one or more numbers. Subscription prices for 2016 are as follows: for paperdelivery,US$890list,US$712.00institutionalmember;forelectronicdelivery,US$784list, US$627.20institutional member. Uponrequest, subscribers topaper delivery ofthis journalare also entitled to receive electronic delivery. If ordering the paper version, add US$10 for delivery withintheUnitedStates;US$69foroutsidetheUnitedStates. Subscriptionrenewalsaresubject tolatefees. Seewww.ams.org/help-faqformorejournalsubscriptioninformation. Eachnumber maybeorderedseparately;please specifynumber whenorderinganindividualnumber. Back number information. Forbackissuesseewww.ams.org/backvols. Subscriptions and orders should be addressed to the American Mathematical Society, P.O. Box 845904, Boston, MA 02284-5904USA. All orders must be accompanied by payment. Other correspondenceshouldbeaddressedto201CharlesStreet,Providence,RI02904-2294USA. Copying and reprinting. Individual readers of this publication, and nonprofit libraries acting for them, are permitted to make fair use of the material, such as to copy select pages for useinteachingorresearch. Permissionisgrantedtoquotebriefpassagesfromthispublicationin reviews,providedthecustomaryacknowledgmentofthesourceisgiven. Republication,systematiccopying,ormultiplereproductionofanymaterialinthispublication is permitted only under license from the American Mathematical Society. Permissions to reuse portions of AMS publication content are handled by Copyright Clearance Center’s RightsLink(cid:2) service. Formoreinformation,pleasevisit: http://www.ams.org/rightslink. Sendrequestsfortranslationrightsandlicensedreprintstoreprint-permission@ams.org. Excludedfromtheseprovisionsismaterialforwhichtheauthorholdscopyright. Insuchcases, requestsforpermissiontoreuseorreprintmaterialshouldbeaddresseddirectlytotheauthor(s). Copyrightownershipisindicatedonthecopyrightpage,oronthelowerright-handcornerofthe firstpageofeacharticlewithinproceedingsvolumes. MemoirsoftheAmericanMathematicalSociety (ISSN0065-9266(print);1947-6221(online)) ispublishedbimonthly(eachvolumeconsistingusuallyofmorethanonenumber)bytheAmerican MathematicalSocietyat201CharlesStreet,Providence,RI02904-2294USA.Periodicalspostage paid at Providence, RI.Postmaster: Send address changes to Memoirs, AmericanMathematical Society,201CharlesStreet,Providence,RI02904-2294USA. (cid:2)c 2015bytheAmericanMathematicalSociety. Allrightsreserved. (cid:2) ThispublicationisindexedinMathematicalReviewsR,Zentralblatt MATH,ScienceCitation Index(cid:2)R,ScienceCitation IndexTM-Expanded,ISI Alerting ServicesSM,SciSearch(cid:2)R,Research (cid:2) (cid:2) (cid:2) AlertR,CompuMathCitation IndexR,Current ContentsR/Physical, Chemical& Earth Sciences. ThispublicationisarchivedinPortico andCLOCKSS. PrintedintheUnitedStatesofAmerica. (cid:2)∞ Thepaperusedinthisbookisacid-freeandfallswithintheguidelines establishedtoensurepermanenceanddurability. VisittheAMShomepageathttp://www.ams.org/ 10987654321 201918171615 Contents Chapter 1. Introduction 1 1.1. Higher order Bohr problem 1 1.2. Higher order almost automorphy 4 1.3. Further questions 9 1.4. Organization of the paper 10 Chapter 2. Preliminaries 11 2.1. Basic notions 11 2.2. Bergelson-Host-Kra’ Theorem and the proof of Theorem A(2) 12 2.3. Equivalence of Problems B-I,II,III 13 Chapter 3. Nilsystems 17 3.1. Nilmanifolds and nilsystems 17 3.2. Nilpotent Matrix Lie Group 19 Chapter 4. Generalized polynomials 21 4.1. Definitions 21 4.2. Basic properties of generalized polynomials 22 Chapter 5. Nil Bohr -sets and generalized polynomials: Proof of Theorem B 27 0 5.1. Proof of Theorem B(1) 27 5.2. Proof of Theorem B(2) 32 Chapter 6. Generalized polynomials and recurrence sets: Proof of Theorem C 43 6.1. A special case and preparation 43 6.2. Proof of Theorem C 46 Chapter 7. Recurrence sets and regionally proximal relation of order d 55 7.1. Regionally proximal relation of order d 55 7.2. Nil Bohr -sets, Poincar´e sets and RP[d] 56 d 0 7.3. SG -sets and RP[d] 60 d 7.4. Cubic version of multiple recurrence sets and RP[d] 64 7.5. Conclusion 69 Chapter 8. d-step almost automorpy and recurrence sets 71 8.1. Definition of d-step almost automorpy 71 8.2. Characterization of d-step almost automorphy 72 Appendix A 75 A.1. The Ramsey properties 75 A.2. Compact Hausdorff systems 76 iii iv CONTENTS A.3. Intersective 78 Bibliography 81 Index 85 Abstract Two closely related topics: higher order Bohr sets and higher order almost automorphy are investigated in this paper. Both of them are relatedto nilsystems. Inthefirstpart,theproblemwhichcanbeviewedasthehigherorderversionof an old question concerning Bohr sets is studied: for any d ∈ N does the collection of {n ∈ Z : S ∩ (S − n) ∩ ... ∩ (S − dn) (cid:5)= ∅} with S syndetic coincide with that of Nil Bohr -sets? It is proved that Nil Bohr -sets could be characterized d 0 d 0 via generalized polynomials, and applying this result one side of the problem is answeredaffirmatively: foranyNil Bohr -setA, thereexistsasyndeticsetS such d 0 that A ⊃ {n ∈ Z : S ∩(S −n)∩...∩(S −dn) (cid:5)= ∅}. Moreover, it is shown that the answer of the other side of the problem can be deduced from some result by Bergelson-Host-Kra if modulo a set with zero density. In the second part, the notion of d-step almost automorphic systems with d∈ N∪{∞} is introduced and investigated, which is the generalization of the classical almost automorphic ones. It is worth to mention that some results concerning higherorderBohrsetswillbeappliedtotheinvestigation. Foraminimaltopological dynamical system (X,T) it is shown that the condition x ∈ X is d-step almost automorphic can be characterized via various subsets of Z including the dual sets of d-step Poincar´e and Birkhoff recurrence sets, and Nil Bohr -sets. Moreover, it d 0 turns out that the condition (x,y) ∈ X ×X is regionally proximal of order d can also be characterized via various subsets of Z. Received by the editor January 14, 2014 and, in revised form, April 8, 2014 and May 24, 2014. ArticleelectronicallypublishedonDecember11,2015. DOI:http://dx.doi.org/10.1090/memo/1143 2010 MathematicsSubjectClassification. Primary: 37B05,22E25,05B10. Key wordsand phrases. NilpotentLiegroup,nilsystem,Bohrset,Poincar´erecurrence,gen- eralizedpolynomials,almostautomorphy,Birkhoffrecurrenceset,theregionallyproximalrelation oforderd. Huang,ShaoandYeweresupportedbytheNNSF(11225105,11371339,11431012,11571335). The authors are affiliated with the Wu Wen-Tsun Key Laboratory of Mathematics, USTC, Chinese Academy of Sciences and Department of Mathematics, University of Sci- ence and Technology of China, Hefei, Anhui, 230026, People’s Republic of China. E-mails are:[email protected];[email protected];[email protected]. (cid:2)c2015 American Mathematical Society v CHAPTER 1 Introduction In this paper we study two closely related topics: higher order Bohr sets and higher order almost automorphy. Both of them are related to nilsystems. In the first part we investigate the higher order Bohr sets. Then in the second part we study the higher order automorphy, and explain how these two topics are closely related. 1.1. Higher order Bohr problem A very old problem from combinatorial number theory and harmonic analysis, rooted in the classical work of Bogoliuboff, Følner [17], Ellis-Keynes [16], and Veech [52] is the following. Let S be a syndetic subset of the integers. Is the set S−S aBohr neighborhoodof zeroinZ (alsocalledBohr -set)? For the equivalent 0 statements and results related to the problem in combinatorial number theory, group theory and dynamical systems, see Glasner [26], Weiss [54], Katznelson [42], Pestov [47], Boshernitzan-Glasner [11], Huang-Ye [41], Grivaux-Roginskaya [32,33]. Bohr-sets are fundamentally abelian in nature. Nowadays it has become ap- parent that higher order non-abelian Fourier analysis plays an important role both incombinatorial numbertheoryandergodictheory. Relatedtothis, ahigher-order version of Bohr sets, namely Nil Bohr -sets, was introduced in [35]. For the 0 d 0 recent results obtained by Bergelson-Furstenberg-Weiss and Host-Kra, see [4,35]. 1.1.1. Nil Bohr-sets. There are several equivalent definitions for Bohr-sets. Here is the one easy to understand: a subset A of Z is a Bohr-set if there exist m∈N, α∈Tm, and a non-empty open set U ⊂Tm such that {n∈Z:nα∈U} is contained in A; the set A is a Bohr -set if additionally 0∈U. 0 It is not hard to see that if (X,T) is a minimal equicontinuous system, x∈X and U is a neighborhood of x, then N(x,U) =: {n ∈ Z : Tnx ∈ U} contains S −S =: {a−b : a,b ∈ S} with S syndetic, i.e. with a bounded gap (S can be chosenasN(x,U ),whereU ⊂U isanopenneighborhoodofx). Thisimpliesthat 1 1 if A is a Bohr -set, then A⊃S−S with S syndetic. The old question concerning 0 Bohr -sets is 0 Problem A-I: Let S be a syndetic subset of Z, is S−S a Bohr -set? 0 Note that Ellis-Keynes [16] proved that S−S+S−a is a Bohr -set for some 0 a∈S. Veechshowedthatitisatleast“almost”true[52]. Thatis,givenasyndetic set S ⊂ Z, there is an N ⊂ Z with density zero such that (S−S)ΔN is a Bohr - 0 set. Kˇr´ıˇz [44]showed thatthereexistsasubset K ofZwithpositiveupper Banach density such that K −K does not contains S−S for any syndetic subset S of Z. 1 2 1. INTRODUCTION ThisimpliesthatProblemA-Ihasanegativeanswerifwereplaceasyndeticsubset of Z by a subset of Z with positive upper Banach density. A subset A of Z is a Nil Bohr -set if there exist a d-step nilsystem (X,T), d 0 x ∈ X and an open neighborhood U of x such that N(x ,U) =: {n ∈ Z : 0 0 0 Tnx ∈ U} is contained in A. Denote by F the family1 consisting of all Nil 0 d,0 d Bohr -sets. We can now formulate a higher order form of Problem A-I. We note 0 that {n∈Z:S∩(S−n)∩...∩(S−dn)(cid:5)=∅} can be viewed as the common differences of arithmetic progressions with length d+1 appeared in the subset S. In fact, S∩(S−n)∩...∩(S−dn)(cid:5)=∅ if and only if there is m∈S with m,m+n,...,m+dn∈S. Particularly, S−S ={n∈Z:S∩(S−n)(cid:5)=∅}. Problem B-I: [Higher order form of Problem A-I] Let d∈N. (1) For any Nil Bohr -set A, is it true that there is a syndetic subset S of Z d 0 with A⊃{n∈Z:S∩(S−n)∩...∩(S−dn)(cid:5)=∅}? (2) For any syndeticsubset S ofZ, is{n∈Z:S∩(S−n)∩...∩(S−dn)(cid:5)=∅} a Nil Bohr -set? d 0 1.1.2. Dynamical version of the higher order Bohr problem. Some- timescombinatorialquestionscanbetranslatedintodynamicalonesbytheFursten- berg correspondence principle, see Section 2.3.1. Using this principle, it can be shown that Problem A-I is equivalent to the following version: Problem A-II: For any minimal system (X,T) and any nonempty open subset U of X, is the set {n∈Z:U ∩T−nU (cid:5)=∅} a Bohr -set? 0 Similarly, Problem B-I has its dynamical version: Problem B-II: [Dynamical version of Problem B-I] Let d∈N. (1) ForanyNil Bohr -setA,isittruethatthereareaminimalsystem(X,T) d 0 and a non-empty open subset U of X with A⊃{n∈Z:U ∩T−nU ∩...∩T−dnU (cid:5)=∅}? (2) For any minimal system (X,T) and any non-empty open subset U of X, is it true that {n∈Z:U∩T−nU∩...∩T−dnU (cid:5)=∅} is a Nil Bohr -set? d 0 In the next section, we will give the third version of Problem B via recurrence sets. The equivalence of three versions will be shown in Chapter 2. 1A collection F of subsets of Z (or N) is a family if it is hereditary upward, i.e. F1 ⊂ F2 and F1 ∈ F imply F2 ∈ F. Any nonempty collection A of subsets of Z generates a family F(A):={F ⊂Z:F ⊃AforsomeA∈A}. 1.1. HIGHER ORDER BOHR PROBLEM 3 1.1.3. MainresultsonthehigherorderBohrproblem. Weaimtostudy Problem B-I or its dynamical version Problem B-II. We will show that Problem B-II(1) has an affirmative answer, and Problem B-II(2) has a positive answer if ignoring a set with zero density. Namely, we will show Theorem A: Let d∈N. (1) If A⊂Z is a Nil Bohr -set, then there exist a minimal d-step nilsystem d 0 (X,T) and a nonempty open subset U of X with A⊃{n∈Z:U ∩T−nU ∩...∩T−dnU (cid:5)=∅}. (2) For any minimal system (X,T) and any non-empty open subset U of X, I = {n ∈ Z : U ∩T−nU ∩...∩T−dnU (cid:5)= ∅} is almost a Nil Bohr -set, d 0 i.e. there is M ⊂ Z with zero upper Banach density such that IΔM is a Nil Bohr -set d 0 As we said before for d=1 Theorem A(1) can be easily proved. To show The- orem A(1) in the general case, we need to investigate the properties of F . It is d,0 interestingthatintheprocesstodothis,generalizedpolynomials(see§4.1foradef- inition) appear naturally. Generalized polynomials have been studied extensively, seefor exampletheremarkable paperby BergelsonandLeibman[6] andreferences therein. Afterfinishing this paper we even find that it also plays an important role in the recent work by Green, Tao and Ziegler [31]. In fact the special generalized polynomials defined in this paper are closely related to the nilcharacters defined there. We remark that Theorem A(2) was first proved by Veech in the case d= 1 [52], and its proof will be presented in Section 2.2. Let F (resp. F ) be the family generated by the sets of forms GPd SGPd (cid:2)k {n∈Z:P (n) (mod Z)∈(−(cid:3) ,(cid:3) )}, i i i i=1 where k ∈ N, P ,...,P are generalized polynomials (resp. special generalized 1 k polynomials) of degree ≤ d, and (cid:3) > 0. For the precise definitions see Chapter 4. i We remark that one can in fact show that F =F (Theorem 4.2.11). GPd SGPd The following theorem illustrates the relation between Nil Bohr -sets and the d 0 sets defined above using generalized polynomials. Theorem B: Let d∈N. Then F =F . d,0 GPd When d = 1, we have F = F . This is the result of Katznelson [42], 1,0 SGP1 since F is generated by sets of forms ∩k {n ∈ Z : na (mod Z) ∈ (−(cid:3) ,(cid:3) )} SGP1 i=1 i i i with k ∈N, a ∈R and (cid:3) >0. i i Theorem A(1) follows from Theorem B and the following result: Theorem C: Let d∈N. If A∈F , then there exist a minimal d-step nilsystem GPd (X,T) and a nonempty open subset U of X such that A⊃{n∈Z:U ∩T−nU ∩...∩T−dnU (cid:5)=∅}. The proof of Theorem B is divided into two parts, namely Theorem B(1): F ⊂F and d,0 GPd Theorem B(2): F ⊃F . d,0 GPd