Saltwater is absolutely filthy. Researchers have found up to ten million viruses in a single drop of seawater. I couldn't find data for bacteria, but there are likely hundreds of thousands of those also.
Corals also produce many substances and toxins completely unrelated to palytoxin.
Considering the animals you were handling weren't even palythoa, it is much more likely that one of the above was the culprit.
I don't know why our hobby has such a fascination with palytoxin. It's dangerous, to be sure. It's not the only dangerous thing in our tanks, not by a long shot. The personal protective equipment you should be wearing anyway because of the other dangerous things in our tanks would protect you from palythoa toxins, if any are even present.