Complex phenomena such as prolongedly undamped ultra-low frequency oscillation (ULFO) and eigenmode re-excitation are observed in the simulations of hydroelectric power systems. Emphases are put on nonlinearities and mode interactions, which cannot be analyzed by traditional eigen-analysis methods. In order to investigate the mechanism of the evolvement of the nonlinear dynamic process in ULFO, this paper proposes a method to analyze the mode interactions quantificationally. First, a disturbed trajectory is decoupled into a set of time-varying components. Second, transfer matrices of eigenmodes are extracted along the trajectory. Third, consecutive sequences of eigenvalues and trajectories of components are formed by a proposed technique. Based on the decoupled components and transfer matrices, the mechanisms of mode interactions and inheritance relationships between eigenmodes are analyzed. The causes and developments of the above complex phenomena are revealed by the proposed method in a test two-machine system. Meanwhile, the accuracy of the eigenmode matching technique is verified in the New England system.