Pineapple sponge! If it's irritating it too much, you can just scrape it off and the rest will just die off. Otherwise, I'd keep it. They're both filter feeders.
+1 on pineapple sponge and although harmless, get it off the clam before it generates another one or more. Using a razor or pairing knife, cut at the base and drop in sump or discard