There are MANY types of worms that we call 'bristle worms'. Many are harmless, even beneficial members of the clean up crew. Some aren't.
Same with other worms, snails, isopods, copepods... even the 'asterina' starfishes. There are many different species that look similar enough that most of us just can't tell the difference.
IMHO, if you've got worms eating your healthy clams and corals, I do not believe it's the same critter I've got. That doesn't mean that I doubt your observations... just not the same critter.
I've watched my bi-color goatfish pull a fairly large bristleworm from under a rock and toss it right down... I've still got bristleworms, but unless I'm moving rocks around, I rarely see them.