The widespread adoption of renewable energy sources presents significant challenges for power system dispatching. This paper proposes a dynamic optimal power flow (DOPF) method based on reinforcement learning (RL) to address the dispatching challenges. The proposed method considers a scenario where large-scale offshore wind farms are interconnected and have access to an onshore power grid through multiple points of common coupling (PCCs). First, the operational area model of the offshore power grid at the PCCs is established by combining the prediction results and the transmission capacity limit of the offshore power grid. Built upon this, a dynamic optimization model of the power system and its RL environment are constructed with the consideration of offshore power dispatching constraints. Then, an improved algorithm based on the conditional generative adversarial network (CGAN) and the soft actor-critic (SAC) algorithm is proposed. By analyzing an improved IEEE 118-node example, the proposed method proves to have the advantage of economy over a longer timescale. The resulting strategy satisfies power system operation constraints, effectively addressing the constraint problem of action space of RL, and it has the added benefit of faster solution speeds.