It depends. If rare earth, then it could be trouble - it will never show up on ICP or anything, so you just have to hope that the stuff binds to organics and get skimmed out, or just binds forever to some aragonite. For normal magnets, they don't do as much damage as people think - get them out of the tank, but they are not the likely reason why you might be having issues.
This kind of simple magnet in a feeding clip was no issue in my tank and not likely an issue in any others. I took it out, but it had likely beed bad for a long time.