The CIE General Colour Rendering Index is currently the criterion used to describe and measure the colour-rendering properties of light sources. But over the past years, there has been increasing evidence of its limitations particularly its ability to predict the perceived colour quality of light sources and especially some LEDs. In this paper, several aspects of perceived colour quality are investigated using a side-by-side paired comparison method, and the following criteria: naturalness of fruits and vegetables, colourfulness of the Macbeth Color Checker chart, visual appreciation (attractiveness/ preference) and colour difference estimations for both visual scenes. Forty-five observers with normal colour vision evaluated nine light sources at 3000 K, and 36 observers evaluated eight light sources at 4000 K. Our results indicate that perceived colour differences are better dealt by the CIECAM02 Uniform Colour Space. Naturalness is better described by fidelity indices even if they did not give perfect predictions for all differences between LED light sources. Colourfulness is well described by gamut-based indices and attractiveness was found to correlate best with gamut-based indices but also with a preference index or a memory index calculated without blue and purple hues. A very low correlation was found between appreciation and naturalness indicating that colour quality needs more than one metric to describe subjective aspects.