No, I have these all over my sump. They actually help me know how much to feed. If their population starts exploding I am overfeeding. If they start dying I am under feeding.
Overall, these guys are harmless, and even beneficial - but much depends on where they're growing. I had a colony grow inside a CPR HOB overflow, which greatly reduced the flow. The colony thrived, though, because _everything_ going into the sump had to go through that sponge.