You're gambling with hair algae, bubble algae, aptasia, nudibranches, flatworms, and possibly ich cysts. Really any fish disease could be present and infect your livestock. QTing frags gives them time to die off and be visually inspected before introducing to your tank.
When I started I didn't QT frags either, but I did dip them in coralRx and then remove them from the plugs they came with and glue them to my own frag plugs. I eventually switched to Bayer dips and now do full blown QT for several weeks in a separate system.
Good luck!