Toadstools are sometimes known to leak toxins into the water. It’s best to have them downstream of the rest of your corals, so that it is filtered out before running past them. Replacing activated carbon every 3-5 days should keep toxins at bay. Invertebrates, such as anemones, are very sensitive to salinity changes, as well as any other inconsistencies. BTAs expel waste often (if fed often), and will look as you described. However, I haven’t seen my RFA do this. Though, it is not uncommon for coral, or anemones to release waste. On the other hand, you said that the mouth is gaping and it is losing color. Constant brown film being expelled, is a sign of heavy stress. It is most likely expelling zooxanthellae. Some corals do this in response to too much light, causing them to bleach. However, IME, anemones can never seem to get enough light. More than likely, it is swings in water parameters.
one more note. Anemones seem to tolerate ammonia less than a lot of SPS IME, so I’d make sure that it stays at 0.