Can you take advantage of the security benefits of Boot Clock Randomization even when sdwdate is disabled?

I have an Debian installation without kicksecure, but I notice that when I disable NTP services weeks later, the clock goes off by a few seconds, I notice this when the TOTP codes stop working.

is this common? well, that would explain why computers constantly synchronize their time