Time drift on MacOS

Version 17.06.2-ce-mac27 (19124)

Can confirm that there is time drift on the Docker client for Mac. I was able to confirm that restarting the Docker client corrects the time issue. Issue is present across all docker containers and docker run busybox date.

There are quite a few reports of this on previous versions. And it was supposedly fixed, but it is not.

The time being off here was only about a minute prior to restarting the Docker client. I cannot say how far it would have been off had the host been put to sleep. In fact, that may have triggered an update and corrected the issue.

It seems the VM clock emulation is just not accurate. Is the VM not using NTP? Can we not have a cron re-sync regularly, even without a Docker client interruption?

Time is currently set by 2 mechanisms:

NTP via chronyd is supposed to keep things in sync with the host
when the host wakes from sleep -- and the clock is too out-of-sync for chronyd to correct -- there's an explicit hwclock -s

I'm not sure what's allowing it to drift in your case. Perhaps the machine wakes, and runs hwclock -s but chronyd has already given up trying to sync the clocks?

When next in this state, could you log into the VM with:

docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

and then query the NTP state with chronyc e.g.

/ # chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

/ # chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 192.168.65.1                  2   6   377    53    +71us[ +120us] +/-   32ms

@djs55

@djs55 sure - will do. Also, please note that my system never went to sleep. And, I know this to be the case because I just installed OS X 10.13 the day before. But, I also rarely put the system to sleep in general.

I'm also seeing this - output captured below:

% docker --version
Docker version 17.09.0-ce, build afdb6d4

% docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh
/ # chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address
/ # chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0   9     0     -     +0ns[   +0ns] +/-    0ns

27secs of drift:

%docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/date ; date
Thu Oct  5 22:53:50 UTC 2017
Fri  6 Oct 2017 09:54:17 AEDT

Seeing this as well

$ docker --version
Docker version 17.09.0-ce, build afdb6d4

$ date
Thu Oct 26 11:57:14 EDT 2017

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/date
Thu Oct 26 16:01:25 UTC 2017

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0   5     0     -     +0ns[   +0ns] +/-    0ns

I'm also seeing considerable drift of several minutes on macOS 10.13.1 (17B46a). My system is always on, I only set the screens to sleep when I'm away from the computer.

$ docker --version
Docker version 17.09.0-ce, build afdb6d4

$ uname -a
Darwin couchbook.local 17.2.0 Darwin Kernel Version 17.2.0: Sun Oct  1 00:46:50 PDT 2017; root:xnu-4570.20.62~10/RELEASE_X86_64 x86_64

$ last reboot | head -n1
reboot    ~                         Sat Oct 28 20:24

$ uptime
16:17  up 5 days, 20:53, 1 user, load averages: 2,92 3,54 3,87

$ date -u ; docker run busybox date
Fr  3 Nov 2017 15:20:43 UTC
Fri Nov  3 15:26:26 UTC 2017

$ date -u; docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /sbin/hwclock -r
Fr  3 Nov 2017 15:25:11 UTC
Fri Nov  3 15:25:12 2017  0.000000 seconds

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /usr/bin/chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0  10     0     -     +0ns[   +0ns] +/-    0ns

I also noticed that the chrony drift file can't be written because of a missing directory:

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

# cat /etc/chrony/chrony.conf
server 192.168.65.1 trust iburst minpoll 2
makestep 0.5 -1
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
cmdport 0

# stat /var/lib/chrony/chrony.drift
stat: can't stat '/var/lib/chrony/chrony.drift': No such file or directory

# ls /var/lib
dhcpcd  docker  misc

This probably doesn't matter much, since it only affects initial clock drift correction after chronyd restarts.

One interesting thing to note though is that the hwclock in the VM is not drifting, but the system time is drifting from RTC. This indicates that the clock provider is doing a bad job.

Currently it is using the TSC time source, it might make sense to experiment with other providers like hpet or acpi_pm. There are various post on the web with TSC causing trouble on HyperV, os it is likely it affects other hypervisors as well.

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

I'll try experimenting with hpet first and see if that helps and try acpi_pm next.

The clocksource can be changed at runtime:

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh

# echo hpet > /sys/devices/system/clocksource/clocksource0/current_clocksource

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

If your VMs system time currently drifts from the rtc, sync it back after switching the clocksource:

# hwclock --hctosys

I found lots of useful info under the following links:

Same here, with a drift of 16 minutes. Fixed by restarting Docker.

$ docker --version                                                                                                                                                               
Docker version 17.09.0-ce, build afdb6d4

$ docker run --rm --net=host --pid=host --privileged -it justincormack/nsenter1 /bin/sh
/ # chronyc activity
200 OK
1 sources online
0 sources offline
0 sources doing burst (return to online)
0 sources doing burst (return to offline)
0 sources with unknown address

/ # chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.65.1                  0  10     0     -     +0ns[   +0ns] +/-    0ns

/ # date
Thu Nov 23 02:35:33 UTC 2017
/# exit

$ date                                                                                                                                                                           
Thu 23 Nov 2017 10:19:57 AWST

Btw. none of the available clocksources fixed the time drift problem for me.

I'm seeing a pretty consistent drift of about 1-2ms per second. This is pretty consistent. Resetting the clock time just changes the time, it doesn't address the drift.

Same problem on Version 17.09.1-ce-mac42 (21090), Channel: stable, 3176a6af01.

My PostgreSQL container's clock gains several seconds in the future every time I wake up my MacBook Pro from sleep. I love Docker, but having to restart it so often is really annoying...

Let me know if I can help in any way.

@adrienjoly

@adrienjoly - here is a tip: docker run --rm --privileged alpine hwclock -s will reset the time in the docker server. No need to stop and restart.

@buchs

@buchs Thanks for the tip! But isn't there a risk of altering data integrity of my PostgreSQL database running on that container, if I change the time without rebooting?

Hmm, interesting question. If you are timestamping entries, and you turn the docker server clock back, you might have issues. What else can you do in a database to cause problems? Constraints based on time? Maybe the log would turn out funny, if you needed to do a recovery. It all depends on what and how you use the database, I suppose.

You're right, changing the time will cause wrong timestamps in our data. That's not that bad for a development environment.
I was mostly worried about PostgreSQL's internal behavior when changing the time back to the past. (i.e. impact on caching, handling of atomic transactions, etc...)

I'm seeing the same issue using a Python image. Note that some servers refuse to authenticate my client due to the time disparity! At present a 13 second disparity:

$ docker --version
Docker version 17.12.0-ce, build c97c6d6

$ docker run -it --rm python:3-jessie python -c 'import time;print(time.time())' && python3 -c 'import time;print(time.time())'
1516141290.7820508
1516141277.759674

The hwclock workaround does work for me.

@djs55

i would say this solves it for me, looking forward to when it lands in the official release, thanks @djs55! best, mike.

Is this slated for this quarter's stable release?

@timkeeler

@timkeeler the fix should be in edge 18.05.0-ce-mac66 and later and should also be in the next stable. Hope this helps!

I hope so and how much this is a problem for me might just jump to a unstable version to get around these issues.

@tmack8001

@tmack8001 been running this for about 3 weeks or so now and it seems to have fully resolved the issue. I'm going ahead and closing this.

Any idea when this fix is going to hit a stable release? Right now seems to be the longest period in Docker For Mac's history without a stable release!

@ThatLarsGuy

@ThatLarsGuy the next stable Desktop release should coincide with the 18.06 engine release. The edge channel currently has 18.06.0-ce-rc1 but this isn't the final release candidate. So it should be soon, depending on whether serious bugs are found in the next release candidate(s).

Not to kick a dead horse (but also, I will, because why else would I be on this thread). This timing bug is a bit long-lived at this point; seriously, it is years long and there's issue history to back that up. Let's get it released already! 🚀

This is enough of a problem for me that I ended up making a docker image to address it until a real fix gets released. Still in the middle of testing it myself, but so far it seems to work well for me: https://hub.docker.com/r/eshork/docker-mac-chrony/

It's a really simple solution. I'm honestly not sure why this doesn't already exist somewhere else. Perhaps I simply didn't stumble across the appropriate google result...

Either way, if it helps one other person, that makes it worth while.

Release notes for Docker for Mac stable release, version 18.06.0-ce-mac70 (26399) say

Bug fixes and minor changes:

Use Simple NTP to minimise clock drift between the VM and the host. Fixes docker/for-mac#2076

@JohnRGee

@JohnRGee thanks for highlighting this -- the fix has indeed been released on both stable and edge channels. Hopefully this is fixed for everyone. If not, please do open a fresh ticket with a set of diagnostics.

Thanks all for your patience!

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

Navigation Menu

Repository navigation

Time drift on MacOS #2076

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Time drift on MacOS #2076

Description

Activity

djs55 commented on Sep 21, 2017

oojacoboo commented on Sep 21, 2017

jpz commented on Oct 5, 2017

ddollar commented on Oct 26, 2017

felixbuenemann commented on Nov 3, 2017

OscarBarrett commented on Nov 23, 2017

felixbuenemann commented on Nov 23, 2017

ggascoigne commented on Dec 7, 2017

adrienjoly commented on Dec 21, 2017

buchs commented on Dec 23, 2017

adrienjoly commented on Dec 24, 2017

kbuchsicf commented on Dec 31, 2017

adrienjoly commented on Jan 3, 2018

hughsw commented on Jan 16, 2018

33 remaining items

xoba commented on May 16, 2018

timkeeler commented on Jun 8, 2018

djs55 commented on Jun 8, 2018

tmack8001 commented on Jun 29, 2018

oojacoboo commented on Jun 29, 2018

ThatLarsGuy commented on Jul 6, 2018

djs55 commented on Jul 9, 2018

eshork commented on Jul 17, 2018

eshork commented on Jul 18, 2018

shadowspawn commented on Jul 26, 2018

djs55 commented on Jul 26, 2018

docker-robott commented on Jun 25, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions