There is some amazing Micro 4/3 gear these days and the system certainly won't limit you...plenty of fast glass, good flash system, etc. I tried Olympus for a while but couldn't get used to the small size (even though the IQ and functionality was excellent)...and ended going back to Canon mostly because I'm very comfortable with that system (I have shot Canon gear as far back as film in the 80s). I will definitely miss the in-body IS system on the Oly...works great.
I found the Olympus single-shot AF (using the OM-D EM-10 Mark II) to be very fast/accurate and the eye/fast tracking was great. They have a whole line of relatively inexpensive prime lenses that are F/1.8. I've tried the 17, 25 and 45mm and all were extremely sharp and well built. They have a 60mm/2.8 macro that is very good. The 45/1.8 is extremely good...nice bokeh...perfect for kid/family snaps and portraits. The tracking AF on the Oly was not as good...not dSLR territory but not terrible. The video AF was a bit annoying...it would consistently hunt in/out. I mostly just used manual focus for video with the kids as I got tired of the hunting. I think this is better on the more advanced Oly bodies...also might be better on Panasonic (and Pany / Oly lenses are interchangeable)...no experience with those.
If you're into shooting video you might want to consider the Canon M system...the dual-pixel AF excels at video (no hunting like contrast-detect systems). The problem with Canon's mirrorless system is the lack of lenses. Unless you're willing to get the adapter and go to larger EF or EF-S glass the options are very limited. Using the adapter and the larger glass sorta defeats the purpose of the compact M4/3 system IMO. If you're going to do that, IMO it might make mores sense to just go for compact SLR like their SL2. Then you can use EF and EF-S glass with no adapter...and it's still fairly compact...and also has the dual-pixel AF for video shooting. The catch with Canon's video system is that you need IS in the lens if you want it...no in-body IS (at least not for still pics...they're now starting to use a pixel-shift type IS for video in some of their newest bodies...but it's not as effective as the sensor-shift types like Oly/Pany/Sony).
I don't know much about Sony...but their mirrorless bodies do seem to be excellent. The main complaint people seem to have with them is useability...i.e. complex / non-intuitive menus, too many bells-n-whistles. etc. Their native lens selection is not the greatest either....although there are a lot of good third-party manufacturer's making glass for their mount. ..which probably helps mitigate that. From what I have seen their AF performance is very good...sensor/IQ performance also.
Whatever you do, you might consider buying some quality used gear if you're on a limited budget...your $$$ will go a lot further. If you're interested in Canon stuff there is a ton of used gear at FredMiranda.com in the B&S forum and prices on a lot of that stuff have really bottomed-out in recent years. You could pick-up a good body and nice 100mm/2.8 macro for cheap. They have other brands too Nikon, Sony, Oly, Pany, etc....just don't know the other gear as well as I know Canon.
Sorry, I know that was a lot...just sorta poured out...LOL
Best of luck with whatever gear you decide on!