Increasing the penetration of Renewable Energy Sources (RES), e.g. wind and solar, intermittency and volatility of the supply-side are increasing in power systems worldwide. Therefore, the power systems need alternative forms of flexibility potentials to hedge against the intermittent power. District Heating Systems (DHS), especially Heat Pump Systems (HPS), show true potentials to provide demand-side flexibility for power systems. This paper aims to survey the literature on the applications of DHS in power system flexibility. In this way, first of all, the basic structure of the DHS, including the heat resources, thermal units, and thermal storage, is surveyed to give insight into the DHS problem. Afterward, classifying flexibility potentials of the DHS, the conventional and advanced control strategies of the DHS are reviewed comprehensively to investigate the role of heat controllers on power system flexibility. To study the compatibility of the controllers to flexible energy markets, the roles of different control schemes in successive trading floors of the electricity markets, from the energy market to ancillary service markets, are investigated. Finally, the optimization solutions, including mathematical and heuristic approaches, with software tools are reviewed to give readers general insight into the way a DHS problem is modeled and solved.