I think most people who have been in the hobby a while always want more. More space, more corals, more (or bigger) fish. Some will also say they wish they had gone with a bigger tank to start — something that I considered a lot before committing to just a 40-gallon tank.
I personally wouldn't want something I didn't like as much, or rushed into. As far as stability goes, yes, more water volume = greater stability, but I see just as many people with big tanks fighting algae blooms as those with mid-sized aquariums.
Do you have the space in your home to mix large portions of water at once? To store the mixing bins? I like that if I need to, I can handle a 50% water change in one Brute can. In the early months of your tank, you may need to do bigger water changes from time to time as parameters swing.
Your corals don't care about size — they need stable parameters, light, and flow. If you were planning on a 55g, you probably weren't planning on a bunch of fish that wouldn't be appropriately sized for your tank.
The voices in the crowd are always going to be loud.

Just consider what you think will make you happy, what you have the space for, and what you have the energy for. That's what matters. Either way, I think you'll have an awesome tank and I'm excited to see what you do with it!