Effect of surface hybridization on RKKY coupling in ferromagnet/topological insulator/ferromagnet trilayer system

We theoretically investigate the RKKY exchange coupling between two ferromagnets (FM) separated by a thin topological insulator film (TI). We find an unusual dependence of the RKKY exchange coupling on the TI thickness ($t_{TI}$). First, when $t_{TI}$ decreases, the coupling amplitude increases at first and reaches its maximum value at some critical thickness, below which the amplitude turns to diminish. This trend is attributed to the hybridization between surfaces of the TI film, which opens a gap below critical thickness and thus turns the surfaces into insulating state from semi-metal state. In insulating phase, diamagnetism induced by the gap-opening compensates paramagnetism of Dirac state, resulting in a diminishing magnetic susceptibility and RKKY coupling. For typical parameters, the critical thickness in Bi2Se3 thin film is estimated to be in the range of 3-5 nm.


I. INTRODUCTION
In the past decade, topological insulators (TI) have emerged as one of the most attractive topics, in both theory and experiment [1][2][3][4][5][6] . Topological insulators possess surface states with strong spin-momentum locking [1][2][3] , which translates to a large spin-dependent effects such as spin-orbit torques [4][5][6] and quantum anomalous Hall effects 7,8 . Moreover, the spin-locked topological surface states of TI are robust under effects of the time-reversal symmetric impurity scattering. These factors enable TI to be one of the most promising candidates for spintronic devices 9 .
At the same time, ferromagnetic hetero-structures (FMs) are also integral spintronic elements that have been used as a platform for practical spintronic devices, such as spin Hall 10-13 and spin-orbit-based memories 14,15 , topological Hall-based sensors 16 . When two ferromagnets form a spin-valve structure where they are separated by a metal spacer, there is a interlayer exchange coupling between the FMs mediated by the itinerant electron in the metal spacer, which is known as RKKY coupling 17,18 . The RKKY coupling oscillates with spacer thickness between ferromagnetic and antiferromagnetic values, and typically its amplitude decreases with increasing thickness. Recently, the RKKY coupling has been studied in TI systems 19,20 , where the magnetic moments can be in the same surface 19 or belong to different surfaces 20 of the TI film. In the latter case 20 , by assuming unchanged surface states as the thickness changes, the interlayer was shown to behave in the same way as in FM/metal/FM system, i.e., oscillatory damping with increasing TI thickness. However, in TI films a) Electronic mail: elehcs@nus.edu.sg b) Electronic mail: elembaj@nus.edu.sg which are very thin, coupling between surfaces becomes significantly dependent on the film thickness 21,22 and it can critically modify the surface states and their magnetic property 23 .
In this work, we study the interlayer coupling between two FMs separated by a thin topological insulator film 20,24,25 , where thickness-dependence of the surface states is considered. We find that, besides the oscillatory behavior, the coupling amplitude is not monotonically dependent on the thickness, but there is a critical value of thickness at which the coupling amplitude reaches its peak. Below the critical value, the coupling does not increase but rather decreases. These trends are attributed to the phase transition at the critical thickness 23 and strong surface hybridization in thin TI films.

II. THEORY
We consider a trilayer system comprising of a thin topological insulator (TI) film sandwiched by two ferromagnetic films (see Fig. 1). In this structure, both top and bottom Diract surface of the TI film are active, and they are independently coupled to top and bottom FMs, respectively, via sd coupling. Note that in a super-thin film, the top FM can also couple to the bottom TI surface and vice versa, however, for the simplicity such couplings are neglected. The system is described by model Hamiltonian where helicity, is then described by h T I (−k) = −h T I (k). The magnetizations are assumed to align along in-plane directions to assure that the gap opening is solely induced by the surface hybridization, see later. The off-diagonal elements describe the hybridization between top and bottom TI surfaces (inter-surface coupling), which is quantified by the tunneling element ∆. In free-standing TI films, the hybridization opens a gap of 2∆ in the TI surface states. In previous work 23 , it has been shown that the gap can be close or open by applying an appropriate in-plane magnetic field. In our case, the proximity coupling (Jm) can take the role of driving field to adjust the gap, see Fig. 1.
In thick TI films, the hybridization coupling is vanishingly small, meanwhile in the limit of thin TI films, it can be related to the TI thickness t T I as 21,22 with B 1 is a material-dependent parameter. With typical value of B 1 = 0.1 eV nm 2 in Bi 2 Se 3 26,27 , the tunneling element is 0.05 eV for 5 nm film, and can be up to 0.25 eV for 2nm thin film 28 .
Eigenenergies of (1) are given by with where s, τ = ±1 representing spin and hyperbola indices, respectively. The band diagrams are shown in Fig. 1.
To derive the interlayer exchange coupling between the ferromagnets, we apply the RKKY formula given by 17,18 In the above, k F ⊥ is the Fermi momentum in the direction perpendicular to the film, which has typical value of the order of 1/a 0 , with a 0 ≈ 1 nm being the thickness of a quintuple layer. χ(q) is the q-dependent magnetic susceptibility of the TI film.
, where A is the contact potential between electron spin and ferromagnets, S is the spin of the FM spin, V 0 is the atomic volume.
The magnetic susceptibility of the TI thin film is given by the Kubo's formula 29 in which m > (n < ) are for occupied (empty) bands, g s = 2 is the g-factor of electron spin, f 0 (E n,k ) is the Fermi distribution function corresponding to eigen-energy branch E n,k . In addition, the orbital angular momentum can also contribute to the total susceptibility, however, for the simplicity we will ignore this contribution.

III. RESULTS
Substituting the energy in Eq. (3) to (5) and assuming that the Fermi energy E F = 0, the magnetic susceptibility at temperature T = 0 can be evaluated as for parallel magnetizations m 1 m 2 , and for the opposite case m 1 − m 2 . In derivation of the above equations, we have assumed strong exchange limit, i.e., J v F k F . Otherwise, since the interlayer exchange coupling is quadratic in the proximity coupling between electron spin and FM spin (see Eq. 4), the weak FM/TI coupling limit is not interesting. From Eq. 6, it is obvious that the spin susceptibility encounters singularity when ∆ = J, see Fig. 2. In general, any singularity in the susceptibility relates to a second order phase transition 29 , which in this case is the transition between insulating state (∆ > J) and metallic state (∆ < J) at the singularity point 23 . The diminishing susceptibility below the critical thickness is possibly due to the emergence of diamagnetism in the insulating surface state 23  open and the surface states are always in the insulating phase, the singularity is thus avoided.
From Eqs. (4) and (6), (7) the interlayer exchange coupling is readily obtained as for the opposite case m 1 − m 2 , where I 0 = N µ 2 B π 2 . First, we see that the exchange coupling has the oscillatory nature of RKKY-type coupling (see Fig. 3).
However, its amplitude does not monotonically decrease with increasing thickness, but reaches its peak at a thickness t crit = π B 1 /J such that ∆ = J. This trend is in accordance with the behavior of the spin susceptibility discussed above. The compensation between paramagnetic Dirac surfaces and diamagnetic gapped-surfaces leads to the diminishing magnetic susceptibility, and so the diminishing interlayer exchange coupling. For typical parameters of Bi 2 Se 3 thin films, the thickness for maximum interlayer exchange coupling is estimated to be in the range of 3-5 nm.

IV. CONCLUSION
In this work, we have shown that topological insulator films as spacers in spin-valve structures play a fruitful role in mediating the exchange coupling between two ferromagnets. In one hand, the TI film provides electrons for mediating the coupling, which induce stronger coupling for thinner film as the magnetic moment transfer rate is stronger. On the other hand, thin TI film will encounter gap-opening in its surface states due to the hybridization, which in turn suppresses the spin susceptibility and RKKY coupling. Therefore, our work provides a guide to optimize TI-based spin valves for spintronics application.