You can
technically keep them in a rectangular tank,
but I strongly recommend against it, as you need to do some pretty intense flow manipulation to get them suspended
just right so they don't have issues with the bottom/sides/corners and so you don't have too many dead zones. A Kreisel tank (the standard style of tank for keeping jellyfish) is definitely a better way to go for jellyfish. I know people have done DIY Kreisel tanks (basically Kreisel tank inserts that are put into rectangular aquariums), but I've only seen it done on small tanks - so, doing a large DIY Kreisel may be pretty difficult and/or expensive, I don't know for sure.
Also, to sort of answer the other questions in your initial post, I'd recommend figuring out which species of jellyfish you want to keep and researching their specific needs (some may need a chiller while others may need a heater, for example). I'd also recommend looking at general jellyfish care/tank maintenance, as jellyfish aquarium needs are quite different from normal aquarium needs (and I'd really suggest researching jellyfish and their needs prior to deciding if you really want to keep them). The article linked below is a good place to start.
Jellyfish are an exciting new realm in the marine aquarium hobby. Not only are jellyfish aquariums now available for purchase, but creating and building your own system is also an exciting project.
reefs.com
All of that said, personally, I'd say to do something else (like a coldwater tank or something unusual like a giant marine hermit crab tank) with the square tank, but if you really want to you could at least theoretically modify your tank to be jellyfish friendly.