You asked a great question, and the answers should really fall into three categories. 1) corals that just don't make it; 2) corals that grow so aggressively that they become a nuisance; and 3) corals that have such a nasty sting that they wipe out other corals as they grow.
I'll skip
#1, but for #2 I agree with those that mentioned GSP and leather coral. I only recently bought a mushroom...so maybe I'll be regretting that soon (it's doubled in size in just 3-weeks already). For #3 you have to be careful of Acans; they have very long sweepers at night and can melt a neighboring colony in one night. I also have a big War Coral colony that I love, but it is slowly killing and growing over an equally large Psamma. Who kills who is not something you think about when you first start buying frags.