For aesthetics it can be important. The amount of detritus they may contain is likely minimal, or nil for practical purposes. Impacts on flow (per your pictures) is nil. Difficulty of manual removal in tight spots might be improved with acetic acid, or (dilute) muriatic acid, use eye and skin protection, outdoors to avoid respiratory impacts from breathing fumes.
If you add live rock to the tank it may/will likely have similar creatures that may repopulate the tank walls fairly rapidly, so unless they bother you I wouldn't let myself be overly concerned with a rigorous removal effort.
Others wouldn't tolerate the unsightly impact of their adhesion to tank surfaces, hth.
