My tank has a 3-4" deep sand bed all around and I gravel vac every time I do a water change which is roughly every month. I only do the water changes to help replenish key elements that I do not particularly dose.
When you gravel vac you just have to pay attention to the way you do it. If you "stir" the sand bed, you will just send all the toxic dust cloud to go in your tank.
So just stop the flow in your tank and do small sections at a time. I gravel all of my tank besides under the rock work. Just take your time and use a small hose so you can easily control the flow. Your freshly graveled areas should look like little mounds. Its totally save if you do it this way.
The bad way is if you siphon and bulldoze the sand while you are doing it. Just stay vertical in each small section and you will be fine.
As for the killing the beneficial bacteria. Sure, you may kill a few hear and there but it won't be enough to hurt your tank.
As for the Diamond Goby idea, yes I have one too. The problem with the goby and if you never gravel vac'ed your sand is he will stir up the dirty sand and your tank can become cloudy for a few hours. My goby caught me slipping and he did this once and caused a nasty cyano outbreak that I am still battling.
Last ramble, when you do water changes, its good to get dirty water out and replenish it with clean water. Hence the gravel vac. If you just take out clean water and replace it with clean water, its good for elements but not necessarily getting rid of the gunk.
Of course having a real deep sand bed that works and functions is ideal, but in our range of 3-4" that is not ideal.