I've had a 55G, 10G, 13.5G, and currently running a 20G. The 10 and 13.5 were both way too small, and even the 20 is starting to feel too cramped. But the 55G was more work and a bigger footprint than I was really happy with. I think my Goldilocks size is probably around 36-40G, but don't have plans to upgrade any time soon.
The 55 to 10 transfer was super easy. The 13.5 was a new build and then got transferred to the 20, and that was a serious pain because the 20 had to go exactly where the 13.5 was. Both relatively small tanks, but getting everything moved over took around 6 hours. I did not have a great plan for what to don with my stock mid-transfer, because I had to drain most of the 13.5 to move it out from the wall, and couldn't just dump it into the 20 because then I wouldn't be able to move that (was doing the transfer by myself). Everyone got to chill in hardware store buckets for a few hours and came through just fine. Worst part of it was sifting through all of my old sand (did not reuse it) to find all of my nassarius.