In recent years, sub-synchronous oscillation accidents caused by wind power integration have received extensive attention. The recorded constant-amplitude waveforms can be induced by either linear or nonlinear oscillation mechanisms. Hence, the nonlinear behavior needs to be distinguished prior to choosing the analysis method. Since the 1960s, the higher-order statistics (HOS) theory has become a powerful tool for the detection of nonlinear behavior (DNB) in production quality control wherein it has mainly been applied to mechanical condition monitoring and fault diagnosis. This study focuses on the hard limiters of the voltage source converter (VSC) control systems in the wind farms and attempts to detect the nonlinear behavior caused by bi- or uni-lateral saturation hard limiting using the HOS analysis. First, the conventional describing function is extended to obtain the detailed frequency domain information on the bi- and uni-lateral saturation hard limiting. Furthermore, the bi- and tri-spectra are introduced as the HOS, which are extended into bi- and tri-coherence spectra to eliminate the effects of the linear parts on the harmonic characteristics of hard limiting in the VSC control system, respectively. The effectiveness of the HOS in the DNB and the classification of the hard-limiting types is proven, and its detailed derivation and estimation procedure is presented. Finally, the quadratic and cubic phase coupling in the signals is illustrated, and the performance of the proposed method is evaluated and discussed.