I do think that it speeds up. Let's take a table, for instance... one encrusted and with a nice base, which may take a year, the top will expand more in year 3 than it did in year 2. If you get into year 4 without needing to significantly frag it back, then it would be even faster. The trouble with year 4 is that nearly nobody gets here without a massively large tank since you can grow wall-to-wall acropora (we are in the SPS forum) from 1" frag in three years if you are doing stuff right.
From year two to three, you can see/notice growth with the bare eye every day or two.
This can get really hard too since a lot of corals are shipped any more and it can take some of that first year to get over shipping and start to take off - so some of that first year does not really count. Point source lights can cause issues with actual colonies. Lots of people frag and frag which also sets corals back. Chemicals and other treatments can set corals back sometimes, along with bouncing parameters.
Steve Weast's Tank at Oregon Reefs or Copp's can show what can happen to colonies in short order with great lighting, lots of room and good husbandry. They have/had some colonies that in two or three years would barely fit into most tanks on this board.
Here is about 16-18 months worth of growth on a Fox Flame... there was about a 4 month period where my tank did nothing because I treated my Fuge for Red Planaria and treated too much and it browned out my corals and they stopped growing for a while. The first photo was after about three months worth of growth, second one about six months later, then the stagnant period, then about three months after that. The larger photo is about 10" across and 8" tall. It is hard to get consistent growth without spurts when you do something dumb like I did. All in all, 1" to 8x10 frag is not bad for 16-18 months for this coral, which is not a slow grower, but not super fast either. I have about another year for growth and when it hits 15-18 inches, it will need to get fragged up because my tank is not that big: