Maybe the small ones are no problem, but the big ones (don't know if they are a different species) will irritate and kill corals.
If you have picky fish that need a lot of feeding like I do (copperband, leopard wrasses, mandarin, potters angel) the population can really grow.
If your tank is in a fallow period, I assume you aren't feeding the tank, and can probably starve them out.