Dips won't hurt the vermetids. I typically get after them with needle-nose pliers. You have to crush the base, because they can withdraw pretty far. They let out strands of mucus to catch food. You may notice some spider web looking things when you feed.
Reducing food in the column will starve them back. I used to have hundreds, but now I am down to dozens. I suspect it's part of a tank maturing, like a cyano bloom or diatoms.