Can't speak for 'anybody'... but no, I don't think you're crazy.
Filter socks, floss, whatever physical media you might consider... all are good at removing larger particulates from your tank. IF, and only if, you clean them regularly. BRS did a video, you need to change your socks more than weekly. Twice a week... daily, even, for them to really be effective. Otherwise, they trap organic particles, mostly fish poop and food, and hold it in place while it breaks down into particles small enough to flow on through the socks.
BRS didn't talk about it, but one additional thing that I've found. They act as a trap, catching beneficial critters, copepods, mysid and gammarus shrimps, etc. These critters would otherwise flow through the system unimpeded, excepting the few that might get mulched by a pump head.
Add to that the huge maintenance task changing and cleaning these filters, and it's no surprise at all that you might choose such a solution.
I did. I do not use socks, floss, or anything of the sort.
I will make one recommendation. If you don't have socks, you've usually got fairly high amounts of food running into a refugeum or overflow compartment. Get yourself a sump monster. Big emerald crab, sally lightfoot, rock crab that came in as a hitchhiker on someone's live rock... something that can live in that sump and keep accumulations of this gunk from building up. It really does help.