Rotor angle stability (RAS) prediction is critically essential for maintaining normal operation of the interconnected synchronous machines in power systems. The wide deployment of phasor measurement units (PMUs) promotes the development of data-driven methods for RAS prediction. This paper proposes a temporal and topological embedding deep neural network (TTEDNN) model to accurately and efficiently predict RAS by extracting the temporal and topological features from the PMU data. The grid-informed adjacency matrix incorporates the structural and electrical parameter information of the power grid. Both the small-signal RAS with disturbance under initial operating conditions and the transient RAS with short circuits on transmission lines are considered. Case studies of the IEEE 39-bus and IEEE 300-bus power systems are used to test the performance, scalability, and robustness against measurement uncertainties of the TTEDNN model. Results show that the TTEDNN model performs best among existing deep learning models. Furthermore, the superior transfer learning ability from small-signal RAS conditions to transient RAS conditions has been proved.