Everyone does it different and is worried about different things....
I personally don't use gloves but I also don't stick my hands in my tank a lot. I also religiously wash my hands before entering the tank.
For me I'm worried what might be on the gloves I'm using. I have worked in the food industry for over 25 years and "sterile" gloves aren't always "sterile". I have used every kind of glove from nitrile, vinyl, latex, neoprene, even the non powdered ones and they all leave some sort of film on your hands...I don't trust that in my tank.
Its all personal preference but I do agree keep your hands out of your tank as much as possible!