Have the same yellow sponge in my tank.
Doesn't seem to bother anything, and will only climb up a coral skeleton, stops just short of the live tissue, and does not irritate the coral.
I've left it alone for years, it grows very slowly for me. Seems to prefer low light, and high flow areas.