I buy only frags. I feel that allowing them to grow into the tank looks way more natural than plopping in a large colony or two.
That being said - remember that coral growth will be exponential. Each individual coral polyp is it's own animal that will grow by division. 1 becomes 2. 2 becomes 4. etc etc. So although your coral may look like it's hardly growing at all, it is. This is where the patience that is so difficult to contain is so very important.
I started my current SPS dominant 120 reef in 2013 with nothing but frags from the local swap. 3 years later, one of my montiporas had clearly outgrown the tank so I traded it to my LFS for store credit. I had to bring it in in a 40qt cooler.
Sure, some corals grow faster than others. Some will out compete others and begin to grow over/around them. I tyr to let them figure it out for themselves.
Here's a side view of my tank from about a year ago. You can easily see that there is a competition going on!
The red digitata has no grown over about 1/2 of the confusa and will eventually overtake the entire thing. I'll be fragging the confusa long before then so that I can start it out again on the other side of the tank.
Have no fear - give them the appropriate environment and they will, indeed, grow just fine!