At room temperature, conventional DNA polymerases remain active, and they use that time to extend primers wherever they can find partial complementarity — mismatched sites, primer dimers, spurious genomic sequences. By the time the thermocycler reaches the true annealing temperature, the reaction has already produced a population of nonspecific products competing with your intended target.
In routine research, a few extra bands on a gel are an inconvenience. In a diagnostic context, they can mean the difference between a correct result and a false call.

