It seems like you might be both over-filtering and trying to force things a bit too much. Have you been running chemical and biological nutrient controls (GFO, carbon dosing, et al) since you set up the tank?
Are you specifically trying to farm chaeto for its own purpose or why the focus only on that?
Are you trying place most of the focus in the system on the refugium so that it needs to look a certain way?
Is the refugium merely a tool to assist the display tank? What is the exact goal you have in mind?
If your goal is simply to keep the annoying algae out of the display, it sounds like you need to wave a "Mission Accomplished" flag and stop the struggle.
If anything, it sounds like you're over-doing things a little. Maybe consider easing back on the extra PO4 and NO3 removal until there's nothing but occasional activated carbon use, if that?
You might have luck with a different macro aglae or different lighting levels in the refugium, BTW....but you'll have to experiment. I'm pretty sure that chaeto is only favored in a high-nitrate/fast-growth environment.