+1 too early. Everyone struggles with this. I am going through this same thing myself. Everyone says patience is the key. I haven't set up a new tank in probably 5 years so take this for what its worth.
A friend of mine gave me some chaeto and some feather caelerpa that I put in after my diatom outbreak and after I had several fish in the tank. I have 10 relatively small fish in my 75 with a 30G sump. My chaeto was dying off at a pretty good pace for a couple of weeks and I was tanking it out in chunks. I clearly got too much for the tank. Not enough waste. Now here I am a couple of weeks later and the little bit of chaeto I have left is hanging on and actually starting to grow a little. As for the caelrpa, even though I just had a little bit of it, probably just 6 or so leaves, it is actual growing and did better the entire time for whatever reason. I am currently running 14 hours of light in my sump to encourage macro algae growth. I was at 6 hours in my main tank in an effort to keep the algae in my sump. I am now up to 8 hours of light in my main tank. The hair algae is just now starting to take off in my sump. None in my display tank. TBD what will happen over time.