Deep learning of ultrafast pulses with a multimode fiber

Characterizing ultrashort optical pulses has always been a critical but difficult task, which has a broad range of applications. We propose and demonstrate a self-referenced method of characterizing ultrafast pulses with a multimode fiber. The linear and nonlinear speckle patterns formed at the distal end of a multimode fiber are used to recover the spectral amplitude and phase of an unknown pulse. We deploy a deep learning algorithm for phase recovery. The diversity of spatial and spectral modes in a multimode fiber removes any ambiguity in the sign of the recovered spectral phase. Our technique allows for single-shot pulse characterization in a simple experimental setup. This work reveals the potential of multimode fibers as a versatile and multi-functional platform for optical sensing.


INTRODUCTION
Multimode fibers (MMFs) provide diverse degrees of freedom in space, spectrum, polarization, and time, enabling a wide range of applications beyond their traditional role in communication. By manipulating the spatial degrees of freedom, an MMF can operate as a diffraction-limited microscope, 1-4 a high-resolution spectrometer, 5-8 a radio-frequency wave sensor, 9,10 an optical pulse shaper, [11][12][13][14][15] a reconfigurable waveplate, 16 and a tailorable nonlinear element. [17][18][19][20] Previously, we demonstrated that the intensity pattern formed by the interference of guided modes at the output of an MMF could be used to recover the spectral amplitude of input light. [5][6][7][8] Recovering the spectral phase, however, is more challenging because different frequencies do not interfere on a linear detector in a time-integrated measurement.
Here, we propose a nonlinear time-integrated measurement of transmitted light through an MMF to extract the spectral phase of an optical pulse. Two-photon absorption on an array of detectors produces a nonlinear speckle pattern. From the speckle pattern, we can retrieve the relative phase of different spectral components of the pulse because those components interfere in the two-photon absorption process. Nonlinear optical processes have been widely used to characterize ultrafast pulses in the absence of a reference pulse. [21][22][23][24][25] However, many of these self-referenced techniques cannot determine the sign of spectral phase or the direction of time. For example, autocorrelation is commonly used to estimate the pulse width, but it always produces a temporally symmetric trace. Our new technique can resolve the direction of time because an MMF (with uncontrolled bending/twisting) does not keep the symmetry of temporal inversion with phase conjugation. Compared to other pulse measurement methods such as FROG, 22 SPIDER, 23 MIIPS, 24 and PICASO, 25 our scheme has a simple experimental setup without moving parts and allows for single-shot measurement, which is of particular importance when measuring unstable pulse trains.
The complexity of our approach is concentrated on phase retrieval from the nonlinear speckle pattern. Taking advantage of the overwhelming advancements in machine learning and deep neural networks, we employ deep learning for phase retrieval. Artificial neural networks were utilized for phase retrieval in FROG ARTICLE scitation.org/journal/app measurements 26 and demonstrated to outperform other phase retrieval algorithms. 27 FROG measurements based on secondharmonic generation suffer from ambiguity in the direction of time, which would cause instabilities in the training of neural networks, unless an additional constraint was imposed on the pulse shape. Our MMF-based technique does not have such a problem. Previously, deep neural networks were employed for imaging through multimode fibers, 28,29 but a large amount of data was needed for training. It is much easier and less expensive to generate data numerically than experimentally to train the neural networks. 30 However, it is difficult to accurately model a realistic fiber with unknown refractive index fluctuations and micro-bending/twisting to produce highquality numerical data for training. Here, we use the experimentally measured transmission matrix of the multimode fiber to calculate two-photon speckle patterns for training purposes. This hybrid method makes it easy and straightforward to generate numerical data for the specific fiber in the experiment. Moreover, noise in the measurement is included in the training process, making the neural network robust compared to conventional phase retrieval methods. Finally, we combine machine learning with compressive sensing by representing the spectral phases of commonly seen pulses in a sparse basis, greatly reducing the number of parameters that need to be retrieved by the neural network.

PRINCIPAL OF OPERATION
Our scheme relies on the speckle pattern formed at the end of a multimode fiber to provide a unique fingerprint of an optical pulse. Each speckle grain at the distal end provides a different sampling of the pulse. At the input, the pulse excites many guided modes with different propagation constants, and thus, it experiences modal dispersion while propagating through the fiber. At the output, individual speckles are formed by different summations of all spectral components of the pulse, each with a varying amplitude and phase. The transmitted pulse displays distinct stretching and distortions from one speckle to another. The spectral amplitude of the input pulse is extracted from the time-integrated intensity measurement of the output speckle pattern via one-photon absorption on the camera, as done previously in Refs. 5 and 7. The spectral phase is recovered from the time-integrated nonlinear measurement of the speckle pattern via two-photon absorption on a different camera. Since cameras detect all speckle grains in parallel, the amplitudes and phases of all spectral components of a pulse can be extracted with a single-shot measurement. An experimental realization of the proposed scheme is shown in Fig. 1(a).
To use the speckle pattern as the fingerprint of a pulse, we first calibrate the spectral to spatial mapping of the MMF (step index, core diameter = 105 μm, numerical aperture = 0.22, and length = 1.3 m). It requires a full-field measurement of the output light as a function of the input frequency. We use a frequency-tunable laser source, and the transmitted field is measured by off-axis holography in an interferometric setup. 12,16 The incident light is linearly polarized, and one polarization of transmitted light is selected for detection. The complex field transmission coefficients measured at multiple frequencies ω are stored in a transmission matrix T(r, ω), where r denotes the spatial location at the fiber output. T(r, ω) relates the input spectral amplitude A(ω) and phase θ(ω) to the complex output field Eout(r, t) for a fixed incident wavefront, (1)

ARTICLE scitation.org/journal/app
The time-integrated intensity pattern (linear speckle pattern) is independent of the spectral phase θ(ω).
The two-photon absorption pattern I 2 (r) = ∫|Eout(r, t)| 4 dt can be expressed as where ϕ(r, ω) denotes the phase of T(r, ω). The dependence of I 2 (r) on θ(ω) can be used to retrieve the spectral phase of the input pulse. When a random superposition of fiber modes is excited due to the spatial complexity, the transmission matrix T(r, ω) possesses no symmetry. The phases ϕ(r, ω) of its elements are randomly distributed over (−π, π]. If the input pulse is temporally reversed and phase conjugated, the sign of θ(ω) is flipped. Since ϕ(r, ω) remains the same, the phase of the transmitted field θ(ω) + ϕ(r, ω) changes. Consequently, the two-photon speckle pattern I 2 (r) is modified. It is instructive to consider the complementary time domain picture, where due to the complex dynamics in the fiber, the temporal impulse response at each output position r is non-symmetric. Hence, two time-reversed inputs, E in (t) and E in (−t), will yield two different temporal dynamics at the output, resulting in two different speckle patterns I 2 (r). With the experimentally measured transmission matrix, we calculate the two-photon pattern I 2 (r) for the synthesized amplitude and phase in Fig. 1(b). Figure 1(c) presents the change in I 2 (r) when the spectral phase of a simulated pulse has its sign flipped. The relative change ⟨|ΔI 2 |⟩/⟨I 2 ⟩, averaged over r, is 0.13. Hence, the two-photon absorption pattern can eliminate the ambiguity with respect to temporal inversion with phase conjugation.

DEEP LEARNING
As shown in Eq. (3), the mapping from the spectral phase θ(ω) of the input pulse to the two-photon speckle pattern I 2 (r) at the fiber output is nonlinear and complex. It is very difficult to recover θ(ω) from the measured I 2 (r). Conventional phase retrieval algorithms are sensitive to noise in the measurement and, thus, cannot provide a reliable recovery. However, once the fiber transmission matrix is known, it is straightforward to calculate the output speckle pattern for any input pulses with Eq. (3). We deploy a convolutional neural network (CNN) to learn the inverse mapping from the output twophoton pattern to the input spectral phase. With noise incorporated into the network training, the CNN outperforms the standard phase retrieval algorithms. 27 Figure 2(a) is the flowchart of the pulse recovery algorithm. First, the amplitude spectrum A(ω) of the pulse is retrieved from the linear speckle pattern I 1 (r) with Eq. (2). 5,6 Then, we calculate I 2 (r) for various θ(ω) using Eq. (3) with the calibrated T(r, ω) and the recovered A(ω). With these numerical data, we train the

ARTICLE scitation.org/journal/app
CNN for the specific spectrum of the probe pulse. The simulated nonlinear speckle pattern is the input of the CNN, and the predicted spectral phaseθ(ω) is compared to the known phase. Their difference is taken as the MAE (mean absolute error), defined as The error is propagated back through the network to update the weights in the CNN. After the training, a pulse with the same amplitude but an unknown phase is launched into the MMF, and the experimentally measured I 2 (r) is sent to the CNN to predict θ(ω). The temporal field E(t) of the pulse is finally obtained by applying a Fourier transform to the recovered spectral field The basic architecture of a general CNN we adopted is shown in Fig. 2(b). It extracts features of the speckle pattern by convolving it with spatial filters. Many filters are applied to the pattern to obtain an array of feature maps. The most important features are kept by a max-pooling layer and passed to the next convolutional layer. The last few layers of the CNN flatten out all the extracted features and map them to the desired output, i.e., the spectral phase. Specifically, we employ the architecture of Res-Net 18 31 in PyTorch machine learning library. 32 Res-Net 18 is a small CNN architecture with less parameters to avoid overfitting and can be trained faster. The details about the network structure, e.g., the number and size of feature maps in each layer, the size of the convolution kernel, are given in Fig. 3 and Table 1 of Ref. 31. In our case, the number of pixels in the input speckle pattern is 224 × 224 × 3. Factor 3 in the third dimension is just a replication of the pattern for three times because the two-photon pattern is an intensity pattern instead of a RGB image. We add a fully connected layer of 1000 × M at the end of the Res-Net, where M is the total number of unknown coefficients αi and βj in Eq. (4). The weights of the neural network are optimized using Adam 33 for 1000 epochs with the initial learning rate set to 1 × 10 −4 . The learning rate is reduced by a factor of 10 subsequently after 200, 400, and 800 epochs.
To reduce the number of parameters that the CNN needs to predict, we represent the spectral phases in a sparse basis. For a chirped pulse, its spectral phase can be expressed as a polynomial, where ω 0 is the central frequency of the pulse spectrum. The zeroth-order term α 0 is a constant phase, which can be set to 0. The first-order term i = 1 represents a linear phase chirp. α 1 determines the time delay of the pulse, but does not affect the pulse shape or the two-photon pattern, so we set α 1 = 0. We keep the second-, third-, and forth-order terms i = 2, 3, 4, which represent quadratic, cubic, and quartic phase chirps. The higher order terms i ≥ 5 are usually negligible, so we set αi≥5 = 0.
If the signal consists of multiple pulses, the interference of these pulses in the spectral domain produces oscillations. The spectral phase exhibits a discontinuity at every local minimum of the amplitude spectrum. We, therefore, introduce a phase jump βj at the frequency ωj corresponding to the jth local minimum of A(ω), where Θ(ωj) is the Heaviside function with the discontinuity at frequency ωj. The magnitude of phase jump βj is within (−π, π]. With the parameterized spectral phase, the CNN only needs to predict the coefficients αi and βj in Eq. (4).
We numerically generate 10 000 pairs of spectral phases and two-photon patterns, 8000 of which are used for training and the rest for validation. The training takes about 8 h on an 8-GPU AWS cluster. Once the CNN is trained, recovering the spectral phase from an experimentally measured two-photon pattern takes only a few seconds.

NOISE SUPPRESSION
The major difficulty for phase retrieval is the noise in the measurement. Experimentally, there are two main sources of noise: the fiber instability and the camera noise. The integration time of the InGaAs camera and the silicon camera is adjusted when recording the linear and nonlinear speckle patterns so that the signal-tonoise ratio (SNR) exceeds 100 for I 1 (r) and 50 for I 2 (r). With this SNR, the camera noise is negligible. The dominant noise comes from the fiber instability. Since the fiber is not thermally stabilized or mechanically isolated in our experiment, ambient temperature drift and/or external vibrations cause changes in the fiber refractive index. Consequently, the phase of transmitted light changes, and such a change varies from one frequency to another. This means that the fiber transmission matrix during the recording of speckle patterns for unknown pulses differs from the calibrated one. Such difference causes the failure of conventional phase retrieval algorithms.
To account for the fiber instability, we incorporate noise into the synthesized data during the training of the CNN. To evaluate this method, we measure the field transmission matrix of the same fiber twice. With the first transmission matrix (TM1), we generate 10 000 pairs of spectral phases and two-photon patterns to train the CNN. Using the second transmission matrix (TM2), we calculate the two-photon patterns with the spectral phases that have never been seen by using the CNN. This set of data is used to test the trained CNN. By using two measured transmission matrices, we account for fiber instability in time. Typically, the standard deviation of phase difference between the two matrices is about 0.2.
Without accounting for the phase fluctuations in the training of CNN, its prediction has limited accuracy. To illustrate this, we consider a pulse with the amplitude spectrum shown in Fig. 3(b) (green line). The spectral phase θ(ω) is generated by using Eq. (4), with only one phase jump β 1 at 1541.4 nm (red curve). 10 000 phase spectra are synthesized in the range of α 2 ∈ (−0.3, 0.3), α 3 ∈ (−0.05, 0.05), and α 4 ∈ (−0.005, 0.005) and β 1 ∈ (−π, π]. 10 000 two-photon absorption patterns I 2 (r) are calculated with TM1:8000 for training the CNN and 2000 for validation. Training and validation errors are obtained from the corresponding datasets. Figure 3(a) shows the training and validation error during the 1000 epoch training process. The training error is similar to the validation error. The minimum validation error is 0.01. A sample of the recovered spectral phase and the temporal pulse shape in the validation set is shown in Figs. 3

(b) and 3(c).
Since the CNN is trained by using TM1, the prediction of θ(ω) (black dotted curve) from I 2 (r) generated by using TM1 (validation data) is accurate. For the 2000 validation spectra, I 2 (r) are also calculated with TM2 as the test dataset. The prediction of θ(ω) from I 2 (r) generated with TM2 (blue dashed curve) is less accurate. The test error is 0.25, as indicated by the black dotted line in Fig. 3(a) mainly results from the fiber instability, captured in the difference between TM1 and TM2.
To take into account the phase fluctuations in the transmission matrix, we add a random phase noise to the transmission matrix when generating the training data for the CNN. The random phase varies from column to column, each column corresponding to one frequency. It simulates the phase difference between TM1 and TM2, which changes with frequency. We incorporate different random ARTICLE scitation.org/journal/app phases into TM1 to compute the two-photon pattern for every synthesized spectral phase. We tune the standard deviation of the phase noise to minimize the error of CNN. With a standard deviation of 0.47, which is close to the phase fluctuations of the measured transmission matrix, we obtain the smallest error for θ(ω) recovered from I 2 (r) generated with TM2 [with CNN trained with I 2 (r) generated by using TM1]. Compared to the case without noise, the test error is reduced from 0.25 to 0.14, as can be seen from the shift of the black dotted line from Figs. 3(a) and 3(b). The validation error [from I 2 (r) generated with TM1] increases to 0.13, similar to the test error but significantly larger than the training error of 0.02. The gap between the training error and the validation error is the effect of overfitting because the phase noise is not a feature that can be learned by using the CNN. Instead, the CNN is trained to ignore the noise. After the training with noise, the recovered spectral phase and temporal pulse shape in Figs. 3(e) and 3(f) agree well with the synthesized ones.

EXPERIMENTAL DEMONSTRATION
In our measurement scheme, the nonlinear process (twophoton absorption) occurs at the detector, not in the fiber. The propagation of the optical pulse in the multimode fiber must be linear; otherwise, the speckle pattern at the fiber output would vary with the incident pulse energy, making it extremely difficult to extract the temporal pulse shape. Fortunately, optical nonlinearity is weak in a MMF as a large fiber core reduces the energy density. Furthermore, when many guided modes are excited in the fiber, modal dispersion stretches the pulse temporally, lowering the peak power and further reducing nonlinear effects.
In contrast, optical nonlinearity in a single mode fiber (SMF) can be significant for short pulses. We use our new scheme to characterize the femtosecond laser pulses transmitted through a 1-m-long SMF. By varying the pulse energy coupled into the SMF, we can tune the strength of optical nonlinearity, which will distort the pulse shape.
We first test the case of very weak nonlinearity. Nearly, transform-limited pulses from a femtosecond laser (NKT, Onefive Origami) at 1550 nm are sent through the SMF. The transmitted pulses are, then, launched into the MMF. The spectral amplitude of the pulse is reconstructed from the linear speckle pattern at the MMF output. As shown in Fig. 4(a), it agrees well with that measured with an optical spectrum analyzer, validating the accuracy of the spectral amplitude recovery. The spectral phase is parameterized according to Eq. (4). The phase discontinuities vanish since the amplitude spectrum displays no local minimum. The three phase chirp terms (i = 2, 3, 4) have values a 2 = −0.14, a 3 = 0.0089, and a 4 = −0.0097. The recovered spectral phase is nearly constant across the frequency range of significant spectral amplitude. From the recovered spectral amplitude and phase, we reconstruct the temporal amplitude and phase of the pulse, as plotted in Fig. 4(b).

ARTICLE scitation.org/journal/app
To quantitatively estimate the accuracy of reconstruction, we calculate the nonlinear speckle patternĨ 2 (r) using the recovered spectral amplitude and phase. As shown in Fig. 4(e), it bears striking similarities to the experimentally measured one I 2 (r) in Fig. 4(d). Both patterns are normalized so that the maximal intensity is equal to 1. Their difference, given by the standard deviation ϵ = √ ∫ |I 2 (r) −Ĩ 2 (r)| 2 dr, is 0.11. For further confirmation, we measure the pulse transmitted through the SMF with an autocorrelator (Femtochrome FR-103XL). Despite its ambiguity in recovery of a pulse shape, the autocorrelation trace is a useful metric to evaluate the accuracy of the pulse shape retrieved with other methods. We calculate the autocorrelation trace of the recovered pulse, and it agrees well with the measured one in Fig. 4(c). In our extensive simulations of numerous pulses, we have never seen two pulses with identical spectral amplitude, but a distinct spectral phase can produce the same two-photon speckle pattern and the same autocorrelation trace. Therefore, the excellent agreements obtained for both the nonlinear speckle patterns and the autocorrelation traces validate the recovered pulse shape.
By coupling more power into the SMF, we enhance the nonlinear processes such as self-phase modulation, cross-phase modulation, and four-wave mixing, which cause spectral and temporal distortions of the pulse. The amplitude spectrum of the pulse transmitted through the SMF is reconstructed with the linear speckle pattern, and it agrees well with the measurement by the OSA in Fig. 5(a). The spectrum is severely distorted from that in Fig. 4(a), and it features three local minima (marked by arrows). When synthesizing the spectral phase, we only consider the phase discontinuity at λ ≃ 1541.4 nm because the other two local minima at 1530.6 nm and 1568.8 nm are too shallow to affect the temporal pulse shape. The recovered spectral phase, plotted by the blue dotted line in Fig. 5(a), exhibits a phase jump of 1.75 at λ = 1541.4 nm. The temporal amplitude and phase of the pulse are, then, obtained by the Fourier transform of the recovered spectral field and plotted in Fig. 5(b). The pulse is asymmetric and has a side lobe. The autocorrelation trace for the recovered pulse has good agreement with that measured with the autocorrelator in Fig. 5(d).
By changing the sign of the spectral phase, the temporal field is inversed. As shown in Fig. 5(e), the side lobe is moved from the front to the tail of the main pulse. While the autocorrelation trace remains the same, the nonlinear speckle pattern in Fig. 5(c) changes. In Fig. 5(f), we plot the intensity over a cross section of the pattern [white dashed line in Fig. 5(c)] for the recovered pulse and the time-inversed pulse. They display significant differences, allowing the CNN to differentiate between the pulse and its time-inversed copy.

DISCUSSION AND CONCLUSION
In summary, we demonstrate a novel method of characterizing spectral phases of ultrafast pulses with a multimode fiber (MMF). The propagation of the pulse in the MMF remains linear and is calibrated with a field transmission matrix. The nonlinear process (two photon absorption) at the MMF output induces interference of different spectral components in the pulse; thus, the nonlinear speckle pattern encodes the spectral phase. The complex interference eliminates the ambiguity in the sign of spectral phase, allowing the direction of time to be recovered. The spectral phases are retrieved with a deep neural network, which is trained with the data numerically synthesized with the experimentally measured fiber transmission matrix and the spectral amplitude recovered from the linear (onephoton absorption) speckle pattern. We combine machine learning with compressive sensing by representing the spectral phase in a sparse basis to dramatically reduce the number of parameters that the neural network predicts. Experimental noise is incorporated into the training process, making the trained network robust against fiber instability.
In our current experiment, the multimode fiber is not thermally or mechanically stabilized; thus, the output speckle patterns change in time due to thermal and mechanical drifts. However, the changes are well behaved: perturbations to the fiber transmission matrix cause only perturbative changes in the output speckle patterns. This is because the fiber output field is linearly related to the transmission matrix, and a small change in the transmission matrix induces a small change in the output field. Thus, both the linear (field intensity) and nonlinear (intensity squared) speckle patterns experience small changes. The recovered pulse shape slightly deviates from the actual one, as confirmed by our numerical simulations. The uniqueness of the nonlinear speckle pattern is essential for our success in training the deep neural network.
We have explored the conventional phase retrieval algorithms that are commonly used for computational imaging and compressed sensing to recover the coefficients in Eq. (4) that describe the spectral phase. However, all these algorithms fail to provide reliable phase recovery from two-photon speckle patterns due to their sensitivities to measurement noise in our experiment. The success of CNN is attributed to its intrinsic ability to filter out noise. For our singleshot pulse recovery scheme, the phase retrieval is more sensitive to noise, due to the lack of averaging over multiple measurements and limited power of the signal in a single measurement; thus, it is essential that the reconstruction algorithm is robust against noise. It is recently shown that the linear-regression-based methods can outperform the deep learning approaches for optical systems with linear input-output relations. 34 In our case, the relation between the input spectral phase and the output two-photon speckle pattern is inherently nonlinear, and a CNN containing nonlinearity functions is well suited for inverse mapping from the two-photon pattern to the spectral phase.
For long-term use of our device, the fiber must be stabilized to avoid fiber movement, thermal and mechanical drifts. In the past few decades, efficient fiber stabilization techniques have been successfully developed for fiber gyroscopes, which may be adopted for our device in the future. In our previous realizations of MMF-based spectrometers, we minimized external perturbations to the fiber by placing it in a metal jacket or in a temperature-controlled chamber. 35 Alternatively, the fiber may be replaced by a multimode waveguide fabricated in a silicon chip, 36 which can be stabilized using a commercial package and temperature controller.
We think the most attractive feature of the multimode fiber based pulse characterization scheme presented here is its simplicity. It does not need a reference pulse, 37,38 allowing stand-alone characterization of ultrafast pulses. Experimentally, it requires only a commercially available multimode fiber and two cameras. The InGaAs camera records the linear speckle pattern for retrieval of spectral amplitude, and the silicon camera records the nonlinear speckle ARTICLE scitation.org/journal/app pattern for spectral phase recovery. While the cost of multimode fibers and the silicon camera is low, the InGaAs camera is expensive and could, therefore, limit adoption of this technique. Since the nonlinear speckle pattern also encodes the information of spectral amplitude of the pulse, it may be used to recover the amplitude spectrum in addition to the phase spectrum. 39 By training a deep neural network to perform an inverse mapping from the nonlinear speckle pattern to the spectral amplitude and phase, it may be possible to characterize a pulse using only the silicon camera, foregoing the need for an expensive InGaAs camera. Of course, the InGaAs camera is still needed for the calibration of the fiber transmission matrix, but this could potentially be performed at the factory if the MMF is thermally and mechanically stabilized and spliced to a SMF to ensure repeatable coupling. 7 By using a silicon camera to measure the two-photon speckle pattern, the current approach is limited to characterizing pulses in the near infrared spectrum (below the bandgap of silicon). However, the general approach of using a multimode fiber and measuring the two-photon speckle pattern could be extended to other spectral regions by first imaging the end of the multimode fiber onto a second-harmonic-generation material and then recording the speckle pattern formed at the second harmonic frequency. In addition, this scheme can be easily tuned to characterize pulses of varying length. Specifically, the temporal resolution and the temporal extent of an optical pulse that can be measured using the MMF technique presented here depend on the spectral resolution of the fiber and the bandwidth over which we calibrate the fiber. The spectral resolution of the MMF employed in the current experiment is 0.24 nm; thus, the temporal range of measurement is 30 ps. The fiber transmission matrix is calibrated in the wavelength range of 50 nm, giving a temporal resolution of 160 fs. By calibrating the fiber over a larger bandwidth, one could measure temporally shorter pulses, while using a longer fiber with finer spectral resolution would enable the measurement of pulses extending over a longer period of time.