It isn't the simplest concept to grasp which is why so many people struggle with it. Yes, the 34V you measured will go away with a ground probe, no issues. Yes, you can get the 34V from properly functioning equipment or faulted equipment but the impact of either would only be felt by you, not your fish. This is where the bird on a wire analogy goes wrong. That analogy works because the air around the bird is an insulator. Marine fish live in a conductive environment. They are more resistive than the water around them so the current will flow around them, not through them. You are more conductive than the air around you, so current will flow through you, not around you. This is why electrofishing doesn't work in the ocean. If it did, this would be how marine fish were collected for this hobby.
There is some research that shows HLLE can be caused by voltage in the water. In this case, it wouldn't matter if it is induced or fault. They are effectively the same to the fish. So, I prefer to run my system at a 0V condition. The other issue is that faulted electrical equipment can leach toxins and copper into the water. That is why I want my GFCI to trip immediately on a fault letting me know I have a problem. Otherwise, you can poison your tank for days or longer and even a GFCI may not trip until you put your hand in the tank.