The rapid growth of vermetids means you have a healthy tank - that's the good part! I have thousands of these things all over my rock - inside the rockwork, growing on the side of frag plugs, all over my rocks and overflow... If you can see my profile avatar, you'll see my Monticap growing all over and around it - it looks pretty cool if you ask me! Gives you an idea of how many of these things I have... I gave up trying to kill them.
Fortunately for me, they don't disturb my coral at all. I see the web's they spit out when my sand bed gets stirred up but my coral have never showed signs of stress by it... They are still growing fine. I wouldn't worry about them "killing" your coral. They are not destructive in my opinion.
They took off when i was feeding phyto or food similar. I suggest just keeping your water column clear of any detritus via filter socks, to not fuel them. Some have starved out and stop growing. While other's that still get a taste of particles thrive and grow... my avatar should be proof enough that they don't kill coral, since it's grown around and over it and keeps growing!
I wouldn't worry too much about it by trying to stick your hands in there each time to kill them. If you see the really big ones, then sure you can break those off but mine haven't gotten as big as some i've seen. They usually just grow about a quarter inch to half an inch at most...