This is a great question.... and one that I spent more time thinking through. I have a 220 g TWV tank. My initial goal was to have it as a mixed reef... but with some nice SPS....and specifically nice acros.
The corals I regret are palys and fuzzy mushrooms. They have absolutely started to take off in my tank. The palys are a nice orange. I have tried nearly everything I can think of to eradicate them. My big fear is to cause a palytoxin issue.... poisoning the tank or myself. I wished I had never ever considered putting them in. I also have some smaller green palys that somehow came in as a hitchhiker. Palys will gain a foothold and start taking off in good reef conditions.
I mentioned green fuzzy mushrooms. I have literally hundreds of them. I have pulled, ripped, removed too many to count. They just grow right back. They are nice for a new tank to get some life in.... and get some color started.... but they again will take off in good reef conditions...and once they get a foothold, they are darn near impossible to eradicate.
One other coral families to be careful of are leathers. I had a huge population of Devils Paws in some of my tanks. In good condition...they will drop buds and reproduce like crazy. I have pulled, ripped and removed. If you leave some materials of foot...they will grow back. Interestingly... I also have some neon toadstools....that I have pulled, ripped and removed that have grown back from nothing now 4 times. I keep giving them away.... and they keep growing back.
I wish some of the acros.... had such a growth habit. Ugh!!!!