It's a matter of personal preference. Some people are very anti plug, and they break everything off the plug and glue it onto their rocks.
I've found that most sps will encrust a plug relatively quickly. So I just glue/epoxy the plug to my rock and let it start growing.
I leave zoas and palys on plugs too. Once you've got about 10-15 polyps, you can't even see the plug anymore. If you start with 3-5 polyp frags, you'll be looking good with 10 in no time.
All my chalices stay on plugs! I want to be able to pick them up for top down shots. They grow nice and flat on plugs so you can frag them easier. As they start to outgrow the plug, you can either epoxy it to a larger plug get a beautiful wedding cake look, or you can start cutting them back and making some money.
I guess what I'm saying is "plugs don't bother me." i've broken a few corals off the plugs either to glue them to larger plugs or to glue them directly to rocks. The superglue seems to break loose from the plug fairly easily with a razor blade.