Outside is probably fine. I wouldn't want any of it in the tank since you'd have micro particles getting into your tank since the melamine foam wears away quickly.
I’ve never had a problem cleaning even the inside of the tank. I’ve always ran a filter sock and in all likelihood, any micro particles ended up there. I never saw any “floaties” roaming around the tank though.