Synthesis and Anisotropic Magnetism in Quantum Spin Liquid Candidates $A$YbSe$_2$ ($A$ = K and Rb)

Quantum spin liquid (QSL) state in rare-earth triangular lattice has attracted much attention recently due to its potential application in quantum computing and communication. Here we report the single-crystal growth synthesis, crystal structure characterizations and magnetic properties of $A$YbSe$_{2}$ ($A$= K, Rb) compounds. The X-ray diffraction analysis shows that $A$YbSe$_{2}$ ($A$ = K and Rb) crystallizes in a trigonal space group, $R$-3$m$ (No. 166) with Z = 3. $A$YbSe$_{2}$ possesses a two-dimensional (2D) Yb-Se-Yb layered structure formed by edged-shared YbSe$_6$ octahedra. The magnetic properties are highly anisotropic for both title compounds and no long-range order is found down to 0.4 K, revealing the possible QSL ground state in these compounds. The isothermal magnetization exhibits one-third magnetization plateau when the magnetic fields are applied in $ab$-plane. Heat capacity is performed along both $ab$-plane and $c$-axis, and features the characteristic dome for triangular magnetic lattice compounds as a function of magnetic fields. Due to the change of the interlayer and intralayer distance of Yb$^{3+}$, the dome shifts to low fields from KYbSe$_2$ to RbYbSe$_2$. All these results indicate the $A$YbSe$_2$ family presents unique frustrated magnetism close to the possible QSL and noncolinear spin states.


I. INTRODUCTION
The quantum spin liquid (QSL) state in geometric frustrated materials attracts wide interests due to the future application of quantum entanglement. 1,2 The highly entangled spins in frustrated magnetic lattices prevent the breaking of any symmetry, even at zero temperature. [3][4][5] Besides the proposed quantum magnets with spin-1/2 transition metal, the rare-earth based materials are also a rich ground for frustrated magnetism. [6][7][8][9][10][11][12] As an example, Yb-based triangular magnetic lattice structures have been proposed as effective spin S eff = 1/2 materials at low temperature due to the crystal electric field (CEF) effect. [13][14][15][16][17][18][19] One Yb 3+ based triangular lattice material, YbMgGaO 4 , has been proposed as a QSL candidate by the magnetization, heat capacity, neutron scattering, muon spin relaxation and thermal conductivity measurements. [20][21][22][23][24][25][26][27][28] YbMgGaO 4 crystallizes in high symmetry (trigonal R-3m) by creating a perfect Yb 3+ triangular magnetic lattice. Non-magnetic Mg 2+ and Ga 3+ ions separate the Yb 3+ triangular magnetic layers by 8.4 Å and the distance between the two Yb 3+ ions within the layer is 3.4 Å, which provides well 2D character for YbMgGaO 4 . However, due to the similar ionic radii of Mg 2+ and Ga 3+ , both Mg 2+ and Ga 3+ tend to mix occupy in the structure. Since Mg 2+ and Ga 3+ have two different charges, this can affect the crystal electric field splitting of the Yb 3+ site compared to the perfect structure. Consequently, this type of intrinsic Mg/Ga disorder between the Yb 3+ layers could perturb the spin liquid ground state. 20,24,25,[29][30][31][32] Therefore, it is important not only to study new Yb 3+ based 2D geometrically frustrated materials, but also to develop rational synthetic pathways targeting novel Yb-based 2D compounds without site disorder, particularly ones that enable the effect of QSL ground state by varying the intralayer and interlayer distances of the Yb 3+ magnetic lattice.
Among the most recently studied Yb 3+ triangular magnetic lattices, the delafossite structure (112-structure type) ARQ 2 (A = Li, Na, K, Rb, Cs, Tl, Ag, Cu; R = rare earth; Q = O, S, Se, Te) attracts widespread attention in search of QSL candidates. [33][34][35][36][37][38] The high symmetry of ARQ 2 structures is reported to have different packing structures, where rare earth ions form 2D triangular lattice. However, depending upon the A-site cation, ARQ 2 structure could crystallize in different space groups. 34 Further, magnetic properties of ARQ 2 compounds can be systematically tuned by either changing Asite cation (interlayer-interactions), number of 4 f electrons in the rare earth ions or by changing the type of Q-site (intralayer interactions). Therefore, ARQ 2 structures serve as an experimental touchstone to identify novel quantum magnetic ground states such as QSL. [33][34][35][36][37]39,40 The presence of S eff = 1/2 ground state of Yb 3+ based NaYbQ 2 structures at low temperature region is proposed by electron spin resonance, magnetization and neutron scattering measurements. [41][42][43][44][45][46][47] Further, no long-range order has been observed in heat capacity at zero field down to mK temperatures at low magnetic fields. 33,[48][49][50][51][52][53][54][55][56] Quantum disordered or QSL ground state is proposed in NaYbO 2 48,51,57 and the signature of the magnetic excitation continuum has been found in NaYbSe 2 from 0.1 to 2.5 meV in inelastic neutron scattering (INS) measurement, suggesting the ground state of NaYbSe 2 is a QSL with a spinon Fermi surface. 58 The continuum-like excitation with shortrange order has been found in CsYbSe 2 , indicating it may be located close to the critical boundary between QSL and 120 • phase. 59 The external magnetic fields could perturb the zerofield magnetic ground state and induce the long-range order where the characteristic 1/3 magnetization plateau can be observed in isothermal magnetization. 49,50 These field-induced phase transitions reveal the unique magnetic phase diagram in Yb 3+ based 112 compounds. 60,61 Moreover, superconductivity has been reported in NaYbSe 2 (T c ∼8 K) under high pressure. 62,63 These variations in the structures and the magnetic properties have motivated us to undertake a comprehensive study on AYbSe 2 (A = K, Rb, Cs), where large A ions could avoid the possible mixed occupancy of A/Yb. In the first step of our investigation in this series, we have studied the magnetic properties of CsYbSe 2 using ac and dc magnetic susceptibility and heat capacity down to 0.4 K. 34,50 Additionally, our INS on CsYbSe 2 single crystals suggest that this system could provide a natural realization of QSL ground state 59 . For this purpose, we extend our study to investigate structure-property correlation of KYbSe 2 and RbYbSe 2 . In this paper, we discuss the single crystal growth, structure characterization and magnetic properties of KYbSe 2 and RbYbSe 2 . To our knowledge, there have not been any reported studies of the magnetic properties of KYbSe 2 and RbYbSe 2 , however, synthesis and the crystal structure of KYbSe 2 has been reported by Gray et al. 64 Single crystal X-ray confirmed that both KYbSe 2 and RbYbSe 2 crystallizes in R-3m space group. The magnetization and heat capacity are performed down to 0.4 K when the applied magnetic field is along the ab-plane and c-axis. Both KYbSe 2 and RbYbSe 2 do not exhibit long-range ordering above 0.4 K suggesting the QSL ground state. Fieldinduced magnetic transitions are found, and heat capacity form a dome shape in the H-T phase diagram. The 1/3 magnetization plateau is observed in both KYbSe 2 and RbYbSe 2 when magnetic field is applied along the ab plane. Finally, this study allows us to establish the empirical relationship between the intralayer/interlayer space of Yb 3+ ions and the magnetic properties of AYbSe 2 (A = Na, K, Rb and Cs) family of compounds.

II. EXPERIMENTAL SECTION
Millimeter-sized hexagonal shape KYbSe 2 and RbYbSe 2 single crystals, synthesized by a two-step method, are shown in Fig. 1(a). In the first step, K or Rb chunk (99.9%, Alfa Aesar), and Yb ingot (99.9%, Alfa Aesar) and Se shot (99.999%, Alfa Aesar) were loaded by the stoichiometric ratio into an alumina crucible and sealed it in a quartz tube. The ampoules were slowly heated to 850 • C and cooled down to room temperature in the furnace. After the first reaction, a red color powder was recovered from the crucibles. In the second step, the resulting powder was loaded into quartz tube with mass ratio 1:10 corresponding flux (KCl or RbCl). After that, tubes were sealed again. The final reaction was 850 • C for 2 weeks. Finally, the reactions were quenched by water. The red platelike single crystals ( Fig. 1(a)) were collected by washing with de-ionized water using suction filtration method. A similar method was employed to synthesize single crystals of both NaYbSe 2 and KLuSe 2 , which were used as references in our measurements.
Energy-dispersive spectroscopy analysis (EDS) was performed using a Hitachi S-3400 scanning electron microscope equipped with an OXFORD EDX microprobe to identify the elemental composition in single crystals and it confirmed the molar ratio of K(Rb):Yb:Se is 1:1:2 as expected. X-ray single-crystal diffraction (SXRD) data were collected using Bruker Quest D8 single-crystal X-ray diffractometer equipped with Mo K α radiation, λ = 0.71073 Å. The data were collected at 200 K using a nitrogen cold stream and crystals were mounted in paratone oil. The crystal diffraction images were collected using φ and ω-scans. The diffractometer was equipped with an Incoatec IµS source using the APEXIII software suite for data setup, collection, and processing. ShelX was utilized for the structural solution and refinement after data reduction via SMART-Plus; spherical absorptions corrections were performed within ShelX after initial refinement. Twinning was taken into account during refinement. 65 Table I provides the detailed unit cell parameters of AYbSe 2 series and the selected bond distances are summarized in Table II. X-ray powder diffraction (PXRD) data was collected on a PANalytical X'pert Pro diffractometer equipped with an incident beam monochromator (Cu K α1 radiation) at room temperature. Diffraction patterns of the flat surface of AYbSe 2 crystals were collected to confirm the 00l orientation (ab-plane), as shown in Fig. 1 Magnetic properties were measured using a Quantum Design (QD) Magnetic Properties Measurement System (MPMS3). The magnetization below 2 K was measured by the MPMS3 iHe3 option. The total mass of the measured crystals was ≈3 mg. Temperature dependent heat capacity was measured in QD Physical Properties Measurement System (PPMS) using the relaxation technique. The mass of the single crystals for heat capacity was ≈1 mg.

A. Crystal Structure
Single crystal X-ray structure investigation reveals that both KYbSe 2 and RbYbSe 2 adopt the trigonal R-3m space group compare to P6 3 /mmc in CsYbSe 2 . The detailed crystallographic information of AYbSe 2 (A = Na, K, Rb, Cs) is given in Table I. The crystal structure is built from 2D Se-Yb-Se layers where A-site cations reside in between the layers. These 2D Se-Yb-Se layers packed along the c-axis as displayed in Fig. 2, however, the packing of Se-Yb-Se layers and A-site cations are different between the R-3m and P6 3 /mmc space groups. The interlayer distance is mainly controlled by the size of the A-site cation. Table II summarizes the bond distances and angles of AYbSe 2 series of compounds, including the interlayer distances. For example, in NaYbSe 2 , the interlayer distance is 6.97 Å while in CsYbSe 2 it is 8.24 Å. This is mainly due to the larger size of the Cs + ion compared to the Na + ion. The interlayer distances for KYbSe 2 and RbYbSe 2 are 7.56 and 7.80 Å, respectively. The 2D Yb-Se-Yb layers are constructed along the ab-plane from edged sharing YbSe 6 units, as displayed in Fig. 2. In both structure types, Yb 3+ ions form an equilateral triangle with Se 2− sitting in the threefold symmetry. It is noteworthy to mention that the size of the Yb 3+ equilateral triangle represent the a-axis of the structures and this is the intralayer distance between the Yb 3+ ions within the 2D lattice. As pointed in Fig. 2, there is a significant variation between the intralayer distances between NaYbSe 2 and CsYbSe 2 . It is 4.057 and 4.148 Å for NaYbSe 2 and CsYbSe 2 , respectively. Additionally, Yb-Se-Yb bond angle also changes significantly throughout the AYbSe 2 series. As pointed in Fig. 2, Yb-Se-Yb bond angles are 91.94 • and 93.73 • for NaYbSe 2 and CsYbSe 2 , respectively. These subtle differences in intra-and interlayer distances, and bond angles clearly effect the magnetic properties of this series of compounds, as we discuss in the next section. The geometrical frustrated Yb 3+ triangular lattices with different intralayer and interlayer distances inspire us to investigate the relationship between the Yb-based 112-typed structures and their magnetism. The orientation is determined by the flat surface, where sharp 00l peaks present in Fig. 1(b). Fig. 3(a-b) present the temperature dependence of magnetic susceptibility χ =M/H, where M is the magnetization and H the applied field, and the inverse magnetic susceptibility is also shown above 2 K along ab-plane and c-axis under 1 T applied magnetic field. No long-range order or anomalies are observed above 2 K. Large anisotropy between H||ab and H||c appears below 50 K, with a magnitude ratio of approximately 3 at 2 K. The temperature dependence of 1/χ exhibits a linear relation at high temperature. When the temperature decreases below 100 K, the curves strongly bend on both directions in KYbSe 2 and RbYbSe 2 , indicating the systems go to the low spin state of Yb 3+ due to the CEF effect. The Curie-Weiss fit of the magnetization at 200 K to 250 K is used in both directions for the two compounds. The Curie Weiss fit including a temperature independent Van Vleck term is used at low temperature region (4-10 K). We present the fitting parameters in Table III. The fitting results show comparable value in KYbSe 2 and RbYbSe 2 on each direction, indicating the similar magnetic interaction of Yb 3+ ions in these two structures. The negative Curie-Weiss temperatures in all measurements indicate the antiferromagnetic interactions between Yb 3+ . The effective moment from high temperature fitting is close to the moment of free Yb 3+ (4.54 µ B ). At low temperature, the effective moments present anisotropic smaller values along these two directions when we consider Van Vleck contribution. Although the precious g values need further electron spin resonance and high magnetic field measurements to confirm, these fitting results at low temperature regions hint at the possible anisotropic g values in these two compounds.
FIG. 2. Crystal structure comparison between the R-3m in NaYbSe 2 and P6 3 /mmc in CsYbSe 2 . In AYbSe 2 (A = Na, K, Rb, Cs) series, replacing Na with larger cations such as K, Rb, and Cs changes the intralayer distances, Yb-Se-Yb bond angles, and the interlayer distances.
The low-T magnetization data of KYbSe 2 is shown in Fig. 4(a) and no evidence of long-range order is observed. However, a broad maximum is found below 2 K, suggesting short-range interactions. This feature is similar to previous report for CsYbSe 2 and NaYbSe 2 , but different from NaYbS 2 . [48][49][50] The maximum becomes broader with increasing the magnetic field. A small jump appears at ∼0.7 K at 2.5 T, which reveals the field-induced long-range magnetic order. The transition temperature increases to maximum ∼1 K with field up to 3 T, then decreases to 0.7 K at 6 T, and finally disappears above 0.4 K at 7 T. The inset of Fig. 4(a) presents the temperature dependence of magnetization under different fields along c-axis. At the low field, the magnetization increases monotonously by decreasing the temperature. However, a maximum appears from 1 to 7 T, suggesting shortrange interactions. Due to the large interlayer distance, the long-range order needs a higher magnetic field than our instrument limit of 7 T. The magnitude at H||c is much smaller than that at H||ab, which proves XY-like interactions between Yb 3+ . These low temperature magnetization results indicate that KYbSe 2 has the possible spin liquid ground state with the effective spin 1/2 state at zero field, and magnetic fields could induce magnetic transitions above 2 T at H||ab. A similar result of RbYbSe 2 is presented in Fig. 4(b), besides the detailed transition temperature shifts to low temperature due to the slight expansion in the structure. The isothermal magnetization of AYbSe 2 (A= Na, K, Rb, Cs) is shown in Fig. 5(a). We could observe all four samples present the similar 1/3 magnetic plateau when magnetic fields are applied in the easy-plane (ab-plane). From the theoretical calculations, the 1/3 magnetic plateau is expected in the easy-plane XXZ model in the quantum picture. 61 The inset of Fig. 5(a) Fig. 5(b). The pink dash line shows the boundary between three states near the plateau. The plateau shifts to the lower fields from NaYbSe 2 to CsYbSe 2 due to the change of intralayer and interlayer interaction. NaYbSe 2 requires the highest magnetic field to excite the plateau at 0.42 K and among the four samples, as it has the shortest distances between Yb 3+ in the triangular layers. KYbSe 2 and RbYbSe 2 follow with the same relation, however, the difference between CsYbSe 2 and RbYbSe 2 is very small. This could be due to the small differences between the interlayer distances of the Rb and CsYbSe 2 structures. Also, we cannot ignore the change of the Yb-Se-Yb stacking sequence between the two structures, which could affect the local environment of YbSe 6 octahedra. Moreover, the two-peak feature in dM/dH of KYbSe 2 and RbYbSe 2 can be observed even at ∼1 K, suggesting the strong spin fluctuation above the long-range order temperature. Heat capacity has been performed on single crystals down to 0.4 K with H||ab and also H||c. The temperature dependence of heat capacity of K/RbYbSe 2 from 0.4 to 200 K at 0 T is shown in Fig. 6(a); key feature is the broad peak from 0.4 to 10 K. This behavior reveals the short-range interactions in the frustrated magnetic layers. No λ anomaly is found above 0.4 K, which is consistent with no long-range order in the magnetization at low fields. The heat capacity of isostructural nonmagnetic KLuSe 2 single crystal is used as the phonon background. The temperature dependence of entropy in K/RbYbSe 2 at 0 T is presented in Fig. 6(b). The magnetic entropy reaches a plateau of ∼5.8 J/mol K above 10 K. The slight deviation at 10 K between KYbSe 2 and RbYbSe 2 may be caused by slightly different interactions between Yb 3+ or phonon contribution. This value is close to Rln2 of spin 1/2 system, confirming the effective spin 1/2 of Yb 3+ in this family. The heat capacity is also measured when magnetic field is applied along ab-plane and c-axis for both samples, as shown in Fig. 6(c-f). In these measurements, the field-induced magnetic transition appears and increases to the highest temperature ∼ 1 K with the largest peak at the moderate field, then decreases to lower temperature with higher fields. Comparing to the transition at H||ab, a much higher field is required at H||c due to the easy-plane feature. There are long tails above the transition temperature, indicating the strong spin fluctuations from Yb 3+ . The shape of the transition anomaly is comparable to the previous reports in NaYbSe 2 , and much sharper than in CsYbSe 2 . 49,50 It could be due to the spin correlations in the different space groups of AYbSe 2 , where more 2D character can be found in CsYbSe 2 compare to the rest of the compounds in AYbSe 2 series.
Based on the heat capacity results, we calculate the entropy for all fields in both samples and plot with the applied magnetic fields in Fig. 6(g-h). We use power law fitting to estimate the magnetic entropy between 0 K and 0.4 K, similar to the previous study in NaYbSe 2 . 49 The entropy releases up to only ∼1.2 J/mol K at the highest transition temperature ∼ 1 K, which is close to 1/5 of Rln2 in spin-1/2 system. We find a quasi-linear relation between the field-induced transition temperature and released entropy at transition temperature, marked as green dashed line in Fig. 6(g-h). Above the long-range order temperature, the release entropy merged to ∼1.9 J/mol K at ∼1.1 K. As the transition temperature varies in a relatively large temperature region with different fields in AYbSe 2 , from ∼0.48 K to ∼0.96 K, much sharper λ anomaly appears and hinting that thermal fluctuation rarely related to the long tails feature. Besides, the released entropy is also independent of magnetic field directions based on the similar entropy vs. temperature behavior of H||ab and H||c in K/RbYbSe 2 , although the field-induced spin states are different. We also compare the result of the previously reported CsYbSe 2 and find similar relation. 50  (e-f) The highest field induced transition temperature and corresponding magnetic field versus inversed intralayer distance and interlayer distance in NaYbS 2 and Na/K/Rb/CsYbSe 2 .
At last, we make the H-T phase diagram of KYbSe 2 and RbYbSe 2 based on the magnetization and heat capacity results, shown in Fig. 7(a-d). The obtained transition temperatures from the magnetization and heat capacity are consistent in both samples. When the fields are applied in the abplane, KYbSe 2 presents dome shape long-range order, separated into three different states by the solid green diamonds (the boundaries of the plateau), which can be recognized as 120 • , collinear up-up-down and 2:1 coplanar spin arrangements. When the magnetic field is applied along c-axis, a similar dome feature is found at a higher field due to the easy-plane feature. The similar phase diagram of RbYbSe 2 is shown in Fig. 7(c-d). The half solid diamonds above the longrange order indicate the strong spin fluctuation. The width between the plateau boundaries is almost independent with the temperature, suggesting thermal fluctuation is barely related to the plateau. We could find the domes shift to the lower field from KYbSe 2 to RbYbSe 2 along both ab and c direction. This is due to the large intralayer and interlayer distance in RbYbSe 2 .
Considering the previous experimental results of AYbSe 2 and this present work, the field-induced transitions present clear anisotropic behavior in this family. This could be re-lated to the intralayer and interlayer distance of magnetic Yb 3+ ions. Based on this assumption, we obtain each highest transition temperature T m with the corresponding field H m in NaYbS 2 and Na/K/Rb/CsYbSe 2 series, 49,50,52 and plotted them against the inversed intralayer distance (1/d intra ) of Yb 3+ and interlayer distance (1/d inter ), as shown in Fig. 7(e-f). Since the similar S eff =1/2 state of Yb 3+ is in this family, the interaction energy should directly relate to the magnetic fields and the temperature. Therefore, H m and T m present similar behavior against structure parameters in these systems. As displayed in Figure 7(e), both H m and T m increase monotonously with 1/d intra or 1/d inter . It is noteworthy to point that, with the smallest value of 1/d intra or 1/d inter in this family, CsYbSe 2 largely deviates from the rest of the structures, Fig 7(e-f). This behavior further implies that intralayer and interlayer interactions between Yb 3+ triangular magnetic lattice of CsYbSe 2 are strongly affected by the size of the Cs + ion and the packing sequence in the overall 2D structure.

IV. CONCLUSION
The KYbSe 2 and RbYbSe 2 single crystals have been synthesized by flux method and characterized by magnetization and heat capacity. There is no long-range order in both samples above 0.42 K at zero field, revealing the possible spin liquid ground state. Large magnetic anisotropy with easyplane feature is found in both samples. Magnetic field application can induce the long-range order along both ab and c crystallographic directions, forming similar dome features in H-T phase diagram. The temperature dependent magnetic entropy indicates the field-induced magnetic transitions and strong spin fluctuations. The 1/3 magnetic plateau is found in each AYbSe 2 compound when the magnetic field is applied in the ab plane. At last, we proposed phase diagrams for K/RbYbSe 2 single crystals and the empirical relation between field-induced transition and intralayer/interlayer distance. All these results indicate the AYbSe 2 family has a effective spin-1/2 frustrated magnetism, which is close to QSL and noncolinear spin states.

ACKNOWLEDGMENTS
The research at the Oak Ridge National Laboratory (ORNL) is supported by the U.S. Department of Energy (DOE), Office of Science, Basic Energy Sciences (BES), Materials Sciences and Engineering Division (MSE). This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the US Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally spon-sored research in accordance with the DOE Public Access Plan(http://energy.gov/downloads/doe-public-access-plan).