That is easy: because distortion pollutes the audio spectrum. Above we see the distortion components produced by a simple distortion mechanism on a single -40dBV sinewave at 1kHz. A zero feedback transistor stage serves as an example of a smooth transfer curve here. Note the original frequency at 0dB in the spectrum. The argument that a smooth distortion mechanism produces distortion components (harmonics) that will be masked by the ear has some ground, but only for single tones. Music does not consist of a single tones however. The second figure shows what happens at the other extreme: here three -40 dBV sinewaves are present simultaneously at 1 kHz, 3.3 kHz and 5.1 kHz and pass through the same smooth distortion mechanism. No amount of masking will be able to hide the terrible mess of distortion products that now emerges.
.
.