Hi, I have a similar sized tank without sump/refugium, I set up my first marine tank in Feb and I admit I have also considered giving up.
I think this hobby is trial by fire, lots don't seem to make it. I have had freshwater tanks for years, and the issues and problems I have faced in those tanks are nothing compared to my marine tank. But if you want to end up with a tank with cool looking corals, fish and inverts, keep your chin up and persevere.
Otherwise send your fish back to your pet store for store credit, empty the whole thing, scrub it thoroughly with distilled water, add gravel and a UG filter and buy some tropical fish with the credit (you will have plenty left over as it's much cheaper). If you make sure to include a good shoal of Corys and a plec or two, you will never hav algae problems again (infact you will need to add algae tablets as they are that good at eating it - and that's not one of those "marine lies"!
But if you don't want give up and you decide to solider on, this is what I'd suggest from a person in a similar situation to you...
1. Add a UV system. This will get rid of algae and bacteria clouding, keep scraping the sides and rear and any free floating algae will get zapped.
It wont fix your diatom issue (annoyingly, I have that issue but it will go eventually as I have silicate remover pouches)
2. Add things to your tank that will help soak up nitrates. Beginner Macro algae like Caulerpa seem a good choice, plus it's an instant addition of greenery to your tank for added colour. But the best looking addition I plan on adding myself soon are Mangroves (pic below not my tank)
3. Why not having a go at acclimating Mollies to saltwater. I have done this in the last few days, 6 of the 8 survived and they go around the tank as a shoal pecking algae like there's no tomorrow. NOTE: they can't tolerate too strong flow, so if you try this make sure you have some low flow chill out zones in your tank.
Eventually, once you have the funds & time, I suggest adding a refugium. I am going to do this as well in a couple months. From everything I have read into them, they provide a lot more options to controlling your main tank. Plus you can have things living and growing in a refugium that would either get eaten or do poorly in the main tank like Cheato & Amphipods... speaking of which has anyone tried making a dark section of the refugium for sponges?
Scott