The existing out-of-step (OOS) protection schemes have proven to be deficient in the prevention of significant outages. OOS protection schemes must not operate in stable power swing, and rapidly isolate an asynchronous generator or group of generators from the rest of the power system in case of unstable power swing. The paper proposes a novel phasor measurement unit (PMU) incorporating a polygon-shaped graphical algorithm for OOS protection of the synchronous generator. The unique PMU-based logic works further to classify the type of swing once the graphical scheme detects it, which can identify the complex power swing produced in the modern power system. The proposed algorithm can take the correct relaying decision in the event of power swing due to renewable energy integration, load encroachment, and transient faults. In this paper, the original and modified Kundur two-area system with a power system stabilizer (PSS) is used to test the proposed algorithm. In the end, it provides assessment results of the proposed relay on the Indian power system during the blackout in July 2012. The results demonstrate that the proposed algorithm is fast, accurate, and adaptive in the modern power system and shows better performance than the existing OOS protection schemes.