I started having a different issue that I think I traced down to the NTP setting.
I have a daily email/text notification set to send out at a certain time each day. I started to get that notification at various times through out the day. When I turned off the NTP setting that issue stopped. This was a couple of months ago and I haven't turned NTP back on see if the issue is still there. I will do that and report back.
When I did have NTP on, the P4 did keep the correct time and did not drift fast that I remember.
There is something not right with whatever the P4 uses to keep internal time, as multiple people are reporting that it's fast. I noticed it sometime back and it's what led me turn on the NTP setting.