I'm in and out of my tank ALL the time... like 1/2 dozen times a day and I don't have any issues. The DT is 7 years old and 125 gallon with no sps... softies and nps are my jam with a few other random lps. I don't immediately wash my hands prior with soap unless I have lotion/cream/food/oil/etc on my hands. I wash my hands a lot throughout the day, so they're usually pretty clean. A small, sps dominant tank, maybe it becomes more problematic putting hands in the tank.
What's actually caused problems is repeatedly brushing against certain corals has given me some sort of skin condition like contact dermatitis that's never really, fully gone away over the past 1-1/2 years (I should probably go the dr. at some point about it). I'll sometimes throw a glove on if I know I need to work around certain corals.