Hard to tell and I don't know enough anyway but, personally, the tube worm things underneath I wouldn't worry about.
What I would worry about is if you get sponges taking hold in between the zoa polyps -- that I would try to brush/pick out now.
*I have one zoa patch that started with one pineapple sponge,,, extracted then a couple more popped up in same place,,, now there's some thick sponge choking out most of the zoa patch from the bottom working it's way up. (I let it get out of hand so don't do that)