Honestly, since you don't want to replace the rock; your best bet is to saw/chisel away the portions that are covered in these...
Even then you're not guaranteed to fully eradicate them.
I know it's not what you wanted to hear, but it is what it is.
These things are poisonous weeds that will spring up anywhere if you let one polyp grow unchecked...