Scientifically, Palythoa species have a gritty/grainy coenenchyme (mat/stalk) and Zoanthus species do not take in the gritty material. That is the biggest actual difference. Size has nothing to do with genus, though can give a rough grouping, though very inconsistent. In many cases, lighting and flow control polyp size to a large degree.