The wide application of power electronic component in power system with renewable energy sources has changed the fault characteristics of conventional power system, resulting in the performance degradation of conventional protection, and even the risk of maloperation and rejection. This brings challenges to the security and stability of the power grid. Therefore, in order to solve these problems, a novel principle of pilot protection based on structural similarity and square error method is proposed. The structural similarity criterion makes use of the difference of fault characteristics between renewable sources and synchronous generators to identify internal faults, while the square error auxiliary criterion is used to solve abnormal calculation of the existing similarity based protection. The detailed model of grid-connected renewable energy power plant is built in Real Time Digital Simulator (RTDS) and the proposed protection algorithm is solidified in the industrial protection device. The effectiveness of the proposed principle is verified by hardware-in-loop dynamic simulation experiments. Compared with conventional differential protection, the proposed protection shows excellent performance in speed and reliability during various faults. In addition, a field short-circuit test is carried out in a real wind farm to verify the effectiveness of the proposed protection scheme. It is proved that this proposed method has the prospect for industrial application.