From all my research the breaking of glass can happen but is often under special circumstances. Caldwell mentioned several specific situations, the most common being a burrowing mantis encountering glass and striking it attempting to enlarge the burrow, causing small chips and leaks.
He also discussed one particular O. scyllarus male who continually struck the same spot of glass in his tanks several times a day, and that particular male ended up breaking two different tanks in one year due to wearing down the glass. From the pictures he posted of the broken glass it appeared to be 1/4" thick, which matched the description that it was a 20g tank. Since you appear to have a good sized O. scyllarus I'd probably opt for 1/4" - 3/8" glass and monitor it to make sure it doesn't exhibit the same behavior if you don't have acrylic.
But this brings me to tank size. I don't think a 5 gallon tank is going to be a suitable long term habitat for that mantis. Something like a 2" N. curacaoensis that spends 90% of its time inside of a rock sure, but even my 3" N. wennerae is always running around and exploring his 15g tank. A medium or large O. scyllarus (5"+) I wouldn't put in anything less than a 30g tank and I'd ensure it has ample live rock, sand, and rubble.