They generally do die out on their own, but it seems it may depend on the exact species; as mentioned in the ReefKeeping.com article above, they sometimes reproduce even in tanks without fish (thankfully it's not common though), so some species may be facultative detritivores/carrion-feeders or able to feed on other critters that come in with live rock (such as other pods, worms, tunicates, etc.).
Anyway, yeah, remove any on the fish, and if you want to try removing as many in the display as possible, there's not really any harm in doing so as long as you don't just leave food in there for them indefinitely - like the article (which covered most of the removal methods available) mentioned, pull any bait before leaving the tank for an extended duration.