Table Of Content

Mott Physics and Topological Phase Transition in Correlated Dirac Fermions Shun-Li Yu1, X. C. Xie2,3,4, and Jian-Xin Li1 1National Laboratory of Solid State Microstructures and Department of Physics, Nanjing University, Nanjing 210093, China 2International Center for Quantum Materials, Peking University, Beijing 100871, China 3Institute of Physics, Chinese Academy of Sciences, Beijing 100190, China and 4Department of Physics, Oklahoma State University, Stillwater, Oklahoma 74078 (Dated: January 6, 2011) We investigate the interplay between the strong correlation and the spin-orbital coupling in the Kane-Mele-Hubbard model and obtain the qualitative phase diagram via the variational cluster approach. We identify, through an increase of the Hubbard U, the transition from the topological band insulator to either the spin liquid phase or the easy-plane antiferromagnetic insulating phase, depending on the strength of the spin-orbit coupling. A nontrivial evolution of the bulk bands in 1 the topological quantum phase transition is also demonstrated. 1 0 PACSnumbers: 03.65.Vf,71.27.+a,71.10.Pm,71.30.+h 2 n In recent years, a new field has emerged in condensed the TBI phase is unstable against the magnetic order- a J matterphysics,basedontherealizationthataspin-orbit ing phase [17, 20, 21]. But the whole phase diagram of interactioncanleadtotopologicallyinsulatingelectronic the KMH model, especially the transition between the 5 phases [1, 2]. A topological band insulator (TBI) has TBI and the MI, and the nature of the single-particle ] a nontrivial band structure resulting from the strong excitations in the bulk and on the edges are still open l e spin-orbit coupling. Theoretical and experimental stud- theoretical questions. As the existence of gapless edge - ieshavefoundsuchmaterialsinbothtwo(2D)[3–5]and states is the direct manifestation of the topological na- r t three (3D) [6–10] dimensions. A common property of ture,thestudyofthesingle-particleexcitationspectrais s . TBI is that it has a charge excitation gap in the bulk, the natural way to investigate the phase transition be- t a butwithgaplesshelicaledge(orsurface)statesprotected tween TBI and MI. Here, we use the (zero temperature) m by the time reversal symmetry lying inside the bulk in- variational cluster approach (VCA) [22], which goes be- - sulating gap. As a new quantum state, which is the Z - yond the mean field theory and takes into account ex- d 2 gradedtopologicaldistinctionfromotherconventionalin- actly the effects of short-range correlations by an exact n o sulators, it has attracted great attention. Though great diagonlizationoftheseparativeclusters. Wefindatopo- c progresshasbeenachieved,thecurrentresearchesmostly logical quantum phase transition from TBI to MI with [ focusontheweaklyinteractingsystems. Ithasbeenpro- increasing U and this process shows a nontrivial evolu- 1 posed that the topological insulator may also appear in tion. StartingfromTBI,thespin-orbitcouplinggap∆ SO v the systems with substantial electron correlations, such closes first and then the Mott gap opens up but without 1 as 4d and 5d transition metal oxides [11, 12]. And the the gapless edge states for increasing U, which is closely 1 electroninteractioneffectplaysacrucialroleindetermin- related to the topological properties of the system. The 9 0 ing the ground state of topological insulators in the 2D closing process of ∆SO driven by the correlations is ac- . limit [13]. Therefore, the effects of electron correlations companying with a splitting of both the conduction and 1 on the topological insulators present a new challenge. valence bands. In the strong spin-orbit coupling regime, 0 1 the state transiting from TBI is the easy-plane AF Mott 1 The correlation effects in topological insulators can be insulator. In the weak coupling regime, a spin liquid : studied either by interaction-driven topological insula- phase emerges between the TBI and the AF Mott insu- v i tors [14–16] or by introducing interactions to a system lators. Inaddition,wealsofindadecreaseinthevelocity X with a strong spin-orbit coupling [11, 17, 18]. In this of the helical edge states due to the correlations in the r Letter, we investigate the model proposed by Kane and TBI phase. a Mele [3] on the honeycomb lattice for describing a 2D TheKane-Mele-HubbardmodelisdefinedasH =H + 0 topological insulator, and introduce the Hubbard inter- H , where H is the model proposed by Kane and Mele U 0 action to this model to analyze the Mott physics. Re- on the honeycomb lattice as shown in Fig. 1(a) [3], cently, the Hubbard model on the honeycomb lattice have been studied by Meng et al [19] using the quan- H =t (cid:88) c† c +iλ (cid:88) ν c† τz c , (1) tum Monte Carlo (QMC) method, in which a spin liquid 0 iσ jσ ij iσ σσ(cid:48) jσ(cid:48) (cid:104)ij(cid:105)σ (cid:104)(cid:104)ij(cid:105)(cid:105)σσ(cid:48) (SL) phase is found to exist between the semi-metallic (SM) phase and the antiferromagnetically (AF) ordered and H the Hubbard interaction, U Mottinsulator(MI)phaseforarangeoftheon-siteinter- action U. The mean field analysis and QMC simulations (cid:88) H =U n n . (2) U i↑ i↓ for the Kane-Mele-Hubbard (KMH) model reveal that i 2 FIG. 1: (color online) (a) The 6-site cluster tiling (dashed lines) on honeycomb lattice used in our calculations of the FIG. 2: (color online) Qualitative phase diagram of KMH bulk properties with VCA. A and B denote the two inequiv- model. SM,TBI,SLandAFinsulatordenotethesemi-metal, alent sites, a1 and a2 are the lattice unit vectors. (b) The topological band insulator, spin liquid and antiferromagnetic first Brillouin zone. b1 and b2 are the reciprocal-lattice vec- insulator, respectively. Above the white dashed line in the tors. (c) Schematic view of tiling the ribbon for calculating AF insulator phase, the z-term of the AF order disappears. the edge states. Here, two superlattices (rectangle with solid lines)areshown. Eachsuperlatticemakesupoftwoclusters containing 12 sites as separated by the dashed line). thelatticeGreen’sfunctionG(t(cid:48))bytheDysonequation G−1(t(cid:48)) = G−1 −Σ(cid:48)(t(cid:48)). G(t(cid:48)) can be determined via 0 the cluster perturbation theory [26], in which G(cid:48)(t(cid:48)) is Here, (cid:104)i,j(cid:105) and (cid:104)(cid:104)ij(cid:105)(cid:105) denote the nearest neighbor (NN) calculated by the exact diagonalization method and the and the next nearest neighbor (NNN), respectively. λ is intercluster hopping V is treated perturbatively. In mo- thespin-orbitcouplingconstantandτ thePaulimatrices. mentumspace,G(t(cid:48))canbeexpressedintermsofG(cid:48)(t(cid:48)) ν = +1(−1) if the electron makes a left(right) turn to ij and V as G(k,ω)=G(cid:48)(k,ω)[1−V(k)G(cid:48)(k,ω)]−1. get to the NNN site. Others are in standard notation. Tocalculatetheedgestates, weconsiderastripgeom- The VCA is a cluster method of the self-energy func- etry and construct a supercluster which is made of sev- tional approach (SFA) [22], which approximates the self- eral clusters. In Fig. 1(c), for example, we arrange two energy of the original system by the self-energy Σ(cid:48) of clusters (12 sites in one cluster) in y direction to form a an exactly solvable reference system with the same in- supercluster. The Green funciton of the supercluster is teraction term. It has been successfully applied to, for givenas(Gsc)−1 =(G(cid:48))−1−W andW istheintercluster instance,theproblemofcompetingphasesinhigh-T su- c hopping matrix in the supercluster. perconductors [23, 24]. Despite the considerable finite- To test the existence of the possible AF order, we will sizeerrors,theVCAcanpredictthequalitativelycorrect include the following Weiss field, trend for the phase diagram [25]. In VCA, the lattice is (cid:88) tiled into identical clusters (as illustrated in Fig. 1, each Hα =hα (−1)ηic† τα c† , (4) AF AF iσ σσ(cid:48) iσ(cid:48) cluster contains a hexagon in our calculations) and the i reference system is made up of the decoupled clusters. whereη =0or1,wheni∈AorB. Intheabsenceofthe Thesingle-particleparameters(denotedbyt(cid:48))oftheref- i spin-orbit interaction, the spin sector has a SU(2) sym- erence system are optimized according to the variational metry. So, we have hz = hx,y. However, this relation principle. And one can add any Weiss field to study the AF AF is broken down when the spin-orbit interaction is turned symmetry broken phases. For any Σ(cid:48) parameterized as on. In this case, we will calculate the grand potential Σ(cid:48)(t(cid:48)), we have the grand potential: Ω(h ) as a function of hz and hx , respectively. AF AF AF Ω[Σ(cid:48)(t(cid:48))] = Ω(cid:48)(t(cid:48))+Trln[−(G−1−Σ(cid:48)(t(cid:48)))−1] Our main results on the interplay between the Hub- 0 − Trln[−G(cid:48)(t(cid:48))], (3) bard interaction and the spin-orbit coupling are summa- rized in the U −λ phase diagram [Fig. 2]. Let us first where Ω(cid:48)(t(cid:48)) and G(cid:48)(t(cid:48)) are the grand potential and discuss the λ=0 line. In VCA, the existence of the AF Green’s function of the reference system, G is the order can be determined by the hα dependence of the 0 AF free Green’s function without interactions. The phys- grand potential Ω(h ). Fig. 3(a) presents the results AF ical self-energy Σ is given by the stationary point for different Hubbard interactions U. For weak U, such ∂Ω[Σ(cid:48)(t(cid:48))]/∂t(cid:48) = 0. For any t(cid:48), Σ(cid:48)(t(cid:48)) is related to as U = 2t and 4t, Ω(h ) shows a monotonic increase AF 3 FIG. 3: (color online) (a) Ω as a function of h for various AF valuesofU atλ=0. (b)ThedensityofstatesforU =4and λ = 0. (c) and (d): Ω vs h at λ = 0.2t along the z and AF x-directions, respectively. with hz (hz = hx in this case), indicating that no AF AF AF AF order forms in the system. However, for a large U such as U ≥ 6t, a minimum appears at finite hz and AF this minimum moves to lower hz values with increase AF ofU. Therefore, wecaninferthatanAForderexistsfor a large U as expected. Interestingly, when plotting the densityofstates(DOS)forU =4tasshowninFig. 3(b), we find that an obvious Mott gap has opened up around the Fermi energy. This paramagnetic insulating phase is identified as the SL phase as also been found recently by Meng et al using the QMC simulation [19]. Therefore, thesystemwillundergophasetransitionsfromthesemi- metal(SM) to SL and then to AF Mott insulator with the increase of U. Thus, we can reproduce the QMC simulation results calculated for λ=0 [19]. When turning on the spin-orbit coupling, we find that FIG. 4: (color online) A(k,ω) for single-particle excitations theSLphasemaintainsforarangeofspin-orbitcoupling in the bulk [Figs.(a), (c), (e) and (g)] and in a ribbon with uptoλ=0.125t. Ontheotherhand,theAForderisnot the zigzag edges [Figs.(b), (d), (f) and (h)] as illustrated in isotropic. As seen from Figs. 3(c) and (d), no minimum Fig.1(c)atλ=0.1t. ThewhitedashedcurvesinFigs.(c),(e) is found at U = 4t for λ = 0.2t in the hz dependence, and (g) are the mean-field fits discussed in the text. From AF but it can be found in the hx dependence. It indi- the up to down figures, U =0,2t,3t,4t. The inset shows the AF cates that within a range of U, the z-direction AF order U-dependence of the renormalized velocity of edge states at is destroyed once the spin-orbit coupling is present. For λ=0.1t and 0.2t. λ<0.25t,whenincreasingU further,wefindtheappear- ance of the z-term in the AF order eventually. However, forλ≥0.25t,ithasnotbeenfounduptoU =10t. Thus, J = 4λ2/U. Notice that the z term in H favors an- 2 2 in the phase diagram we plot the white dashed-line sep- tiparallel alignment of the spins on the NNN sites, thus arating the AF order with and without the z-term. The it will introduce a frustration to the NN AF correlation easy-planeAForderistheresultoftheinterplaybetween expressed by H . On the other hand, the xy term in 1 the Hubbard interaction and the spin-orbit coupling. As H favors a ferromagnetic alignment, so no frustration is 2 is well known, the NN hopping will generate anisotropic introduced. As a result, the H term coming from the 2 AF Heisenberg term H = J (cid:80) S · S with J = spin-orbit coupling will suppress the z-term of the AF 1 1 (cid:104)ij(cid:105) i j 1 4t2/U in the strong-coupling limit. Similarly, the NNN order. spin-orbit coupling generates an anisotropic exchanging At another limit U = 0, a TBI is expected to occur term H =J (cid:80) (−SxSx−SySy+SzSz) [17], with once the spin-orbit coupling is turned on [3]. The TBI 2 2 (cid:104)(cid:104)ij(cid:105)(cid:105) i j i j i j 4 is characterized by gapless edge states protected by the H(2) =−(J /2)(cid:80) (a† a −a† a )(a† a −a† a ) λ 2 (cid:104)(cid:104)ij(cid:105)(cid:105) i↑ j↑ i↓ j↓ j↑ i↑ j↓ i↓ bulkgapopenedbythespin-orbitcouplinginthesingle- and choose another parameter χ=(cid:104)a† a −a† a (cid:105). By particle spectrum. The spectral function of single parti- i↑ j↑ i↓ j↓ using mA and χ as adjustable parameters, we can give a cles is given by A(k,ω) = −2ImG(k,ω)/π. The results fittothenumericalresults,whichisplotaswhitedashed for several U at λ = 0.1t are presented in Fig. 4, where lines in Fig. 4. This simple fit provides a possible un- the bulk bands are plotted along the lines shown in Fig. derstanding of the gap closing and reopening processes 1(b)andtheedgestatesarecalculatedfromaribbonwith in the bulk. the zigzag edges [Fig. 1(c)]. For U =0, one can see that Finally, let us discuss the possible effect of electron a bulk gap opens resulting from the spin-orbit coupling correlations on the edge states. As shown in the inset [Fig. 4(a)]. At the meantime, clear gapless edge states of Fig. 4, we notice a visible reduction of the velocity in with sizeable spectral weights emerge [Fig. 4(b)]. These helicalDiracfermionsattheedgeintheTBIphase. This results reproduce perfectly the characters of a TBI [3]. renormalization arising from the two-particle scattering In the presence of Hubbard interaction, the VCA cap- between the left and right moving modes due to elec- tures exactly the short-range correlation effects by the tron correlations, which is allowed by the time reversal exact diagonlization of the small clusters used to tile the symmetry [27, 28]. lattice. We find that the bulk gap is reduced firstly and the edge states are stable against a weak U, as shown in In summary, we have investigated the interplay be- Figs. 4(c) to (f). When U is increased further, the bulk tween the Hubbard interaction and the spin-orbit cou- gap closes and the edge states disappear simultaneously. pling in the Kane-Mele-Hubbard model with the varia- Afterthat,abulkgapwiththecharacteroftheMottgap tionalclusterapproach. WemapadetailU−λphasedi- occurs and no edge states reemerge anymore, as shown agram, in which the topological band insulator, the spin in Figs. 4(g) and (h). Thus, we determine the phase liquid,andtheantiferromagneticinsulatorareidentified. boundary where the TBI disappears by a criteria that We have shown a nontrivial evolution of the bulk bands the bulk gap closes and the edge states disappear. Com- in the topological quantum phase transition. biningwiththeresultsdescribedabove, wecanconclude This work was supported by NSF-China and the that the TBI phase will make transition to the SL phase MOST-China. XCX is also supported by US-DOE when λ ≤ 0.125t and to the easy-xy plane AF phase for through the grant DE-FG02-04ER46124. λ>0.125t, as presented in the phase diagram of Fig. 2. According to the bulk-boundary correspondence [1], theexistenceofgaplessedgestatesdependsonthetopo- logical class of the bulk band structure. The transition [1] M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 from TBI (topologically nontrivial state) to MI (toplogi- (2010). cally trivial state) must undergo a gap closing process in [2] X. L. Qi and S. C. Zhang, arXiv:1008.2026 (2010). thebulk. Asfarasweknow,thisprocessisdemonstrated [3] C.L.Kane,andE.J.Mele,Phys.Rev.Lett.95,146802 (2005);C.L.Kane,andE.J.Mele,Phys.Rev.Lett.95, clearly for the first time by a systematic numerical cal- 226801 (2005). culation presented here. [4] B. A. Bernevig, T. A. Hughes, and S. C. Zhang, Science 314, 1757 (2006). A first attempt to understand the evolution of the [5] M.König,.Wiedmann,C.Bru¨ne,A.Roth,H.Buhmann, spectrums in the KMH model is to include the AF or- L. W. Molenkamp, X. L. Qi, and S. C. Zhang, Science der parameter mA = −mB = |(cid:104)n − n (cid:105)| (A and B i↑ i↓ 318, 766 (2007). denote the sublattice in Fig. 1(a)) [17], which is con- [6] D.Hsieh,D.Qian,L.Wray,Y.Xia,Y.S.Hor,R.J.Cava, sidered as a result of the electron correlations. This and M. Z. Hasan, Nature (London) 452, 970 (2008). gives rise to the mean field dispersion given by E(k) = [7] D. Hsieh, Y. Xia, D. Qian, L. Wray, J. H. Dil, F. Meier, ±(cid:112)ε2(k)+(λ−Um/2)2, with ε(k) the bare dispersion. J.Osterwalder,L.Patthey,J.G.Checkelsky,N.P.Ong, A. V. Fedorov, H. Lin, A. Bansil, D. Grauer, Y. S. Hor, When Um/2 = λ, the spin-orbit gap closes. Then an- R.J.Cava,andM.Z.Hasan,Nature(London)460,1101 other gap Um/2−λ with a character of the Mott gap (2009). opens up with the further increase of U. However, [8] Y. Xia, D. Qian, D. Hsieh, L. Wray, A. Pal, H. Lin, A. the evolution shown in Fig. 4 exhibits a more com- Bansil, D. Grauer, Y. S. Hor, R. J. Cava, and M. Z. plex behavior, namely both the valence and conduction Hasan, Nat. Phys. 5, 398 (2009). bands around K are split into two subbands. It im- [9] H. Zhang, C. X. Liu, X. L. Qi, X. Dai, Z. Fang, and S. plies that another interaction term is needed to be in- -C. Zhang, Nat. Phys. 5, 438 (2009). [10] Y. L. Chen, J. G. Analytis, J.-H. Chu, Z. K. Liu, S.-K. cluded. Comparing the numerical results for different U Mo, X. L. Qi, H. J. Zhang, D. H. Lu, X. Dai, Z. Fang, and λ, we note that the band splitting around K de- S. C. Zhang, I. R. Fisher, Z. Hussain, and Z.-X. Shen, pends on λ2/U. This is the exchange integral in H 2 Science 325, 178 (2009). coming from the second-order process of the spin-orbit [11] D. Pesin and L. Balents, Nat. Phys. 6, 376 (2010). interaction as described above. So, we rewrite H as [17] [12] A.Shitade,H.Katsura,J.Kune,X.-LQi,S.-C.Zhang, 2 5 and N. Nagaosa, Phys. Rev. Lett. 102, 256403 (2009). [21] D. Zheng, C. Wu, and G. M. Zhang, arXiv:1011.5858 [13] M.Liu,C.Chang,Z.Zhang,Y.Zhang,W.Ruan,K.He, (2010). L. -l. Wang, X. Chen, J. -F. Jia, S. -C. Zhang, Q. -K. [22] M. Potthoff, Eur. Phys. J. B 32, 429 (2003); M. Pot- Xue, X. Ma, and Y. Wang, arXiv:1011.1055 (2010). thoff, M. Aichhorn, and C. Dahnken, Phys. Rev. Lett. [14] S. Raghu, X. L. Qi, C. Honerkamp, and S. C. Zhang, 91, 206402 (2003). Phys. Rev. Lett. 100, 156401 (2008). [23] D. Sénéchal, P.-L. Lavertu, M.-A. Marois, and A.-M. S. [15] K. Sun, H. Yao, E. Fradkin, and S. A. Kivelson, Phys. Tremblay, Phys. Rev. Lett. 94, 156404 (2005). Rev. Lett. 103, 046811 (2009). [24] M. Aichhorn, E. Arrigoni, M. Potthoff, and W. Hanke, [16] M. Dzero, K. Sun, V. Galitski, and P. Coleman, Phys. Phys. Rev. B 74, 024508 (2006). Rev. Lett. 104, 106408 (2010). [25] M. Balzer and M. Potthoff, Phys. Rev. B 82, 174441 [17] S.RachelandK.L.Hur,Phys.Rev.B82,075106(2010). (2010). [18] C. N. Varney, K. Sun, M. Rigol, and V. Galitski, Phys. [26] D. Sénéchal, and A.-M. S. Tremblay, Phys. Rev. Lett. Rev. B 82, 115125 (2010). 92, 126401 (2004). [19] Z.Y.Meng,T.C.Lang,S.Wessel,F.F.Assaad,andA. [27] C.Wu,B.A.Bernevig,andS.C.Zhang,Phys.Rev.Lett. Muramatsu, Nature (London) 464, 847 (2010). 96, 106401 (2006). [20] M. Hohenadler, T. C. Lang and F. F. Assaad, [28] C.XuandJ.E.Moore, Phys.Rev.B73, 045322(2006) arXiv:1011.5063 (2010).