The energy consumption of buildings accounts for approximately 40% of total energy consumption. An accurate energy consumption analysis of buildings can not only promise significant energy savings but also help estimate the demand response potential more accurately, and consequently brings benefits to the upstream power grid. This paper proposes a novel physical-data fusion modeling (PFM) method for modeling smart buildings that can accurately assess energy consumption. First, a thermal process model of buildings and an electrical load model that focus on building heating, ventilation, and air conditioning (HVAC) systems are presented to analyze the thermal-electrical conversion process of energy consumption of buildings. Second, the PFM method is used to improve the accuracy of the energy consumption analysis model for buildings by modifying the parameters that are difficult to measure in the physical model (i.e., it effectively modifies the electrical load model based on the proposed PFM method). Finally, case studies involving a real-world dataset recorded in a high-tech park in Changzhou, China, demonstrate that the proposed method exhibits superior performance with respect to the traditional physical modeling (TPM) method and data-driven modeling (DDM) method in terms of the achieved accuracy.