24

I have ntpd running on my server. It's all the default settings, except I commented out its ability to be a server to other machines:

# restrict -4 default kod notrap nomodify nopeer noquery                                                                    
# restrict -6 default kod notrap nomodify nopeer noquery   
restrict default ignore

If I run ntpdate -q ntp.ubuntu.com, I'm told that my machine's clock is off by 7 seconds.

What's going on? How can I diagnose what's happening, is there a log I can turn on?

more info #1

# ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 91.189.94.4     193.79.237.14    2 u   30   64    7  108.518   -0.136   0.361

more info #2

Here's what this looked like when I asked the question:

# ntpdate -q ntp.ubuntu.com
server 91.189.94.4, stratum 2, offset 7.191308, delay 0.13310
10 Jan 20:38:09 ntpdate[31055]: step time server 91.189.94.4 offset 7.191308 sec

And here's what it looks like now, after restarting ntpd a couple times (I'm assuming that's what fixed it):

# ntpdate -q ntp.ubuntu.com
server 91.189.94.4, stratum 2, offset 0.000112, delay 0.13164
10 Jan 20:47:03 ntpdate[31419]: adjust time server 91.189.94.4 offset 0.000112 sec

more info #3

I uninstalled ntp and installed openntpd and ran /usr/sbin/ntpd -d, and I'm seeing output like this:

reply from 64.73.32.134: offset 6.715003 delay 0.041152, next query 30s
reply from 208.53.158.34: offset 6.700224 delay 0.036263, next query 31s
adjusting local clock by 6.734120s
reply from 72.18.205.156: offset 6.708575 delay 0.035885, next query 30s
reply from 64.73.32.134: offset 6.701463 delay 0.044199, next query 33s

Which to me pretty clearly indicates that I'm not able to set the time on my server (although, with regular ntp, it does seem to update sometimes...).

more info #4

My VPS provider says:

The latest kernels should not lock your system to our dom0's clock, to be on the safe side you can set xen.independent_wallclock = 1 in your sysctl.conf.

Which I suppose still does not address the issue of the VPS needing a CPU available in order to do correct timing calculations.

John Bachir
  • 2,364
  • 7
  • 29
  • 37
  • Is that your entire config file? If you run `ntpq -np`, what is the output? – David Mackintosh Jan 10 '11 at 20:44
  • Where's the rest of the config? There's no upstream server for your host to get time from. – Aaron Copley Jan 10 '11 at 20:45
  • no, it's not my entire config file, it's just the part i changed from the vendor config file. editing my question to show the output if `ntp -np` – John Bachir Jan 10 '11 at 20:45
  • 7
    Got it. It looks like ntpd was working normally. NTPd will "slew" your clock back into sync gradually. A sudden change in time can cause big problems for certain running processes so NTP works by speeding up or slowing down the length of a second to gradually make adjustments. – Aaron Copley Jan 10 '11 at 20:50
  • My server has been running for many months, so I don't think slewing is the issue -- unless every time I reboot it gets reset to the underlying hardware's clock or some such thing. – John Bachir Jan 10 '11 at 20:52
  • 1
    Yes, the kernel will start with the hardware clock on boot since on boot that is it's only reference. If it's been running for many months, as you say, then that isn't it. You can tell NTP to sync to your hardware clock. I'm not sure about Ubuntu but on Red Hat based systems that's in /etc/sysconfig/ntpd. You can look there or refer to your hardware's documentation. – Aaron Copley Jan 10 '11 at 20:56
  • 1
    I also don't think you understand that ntpdate is a standalone application. It doesn't have anything to do with ntpd and shouldn't be used to troubleshoot it. The reason that ntpq was suggested with the -p options to show peering. If ntpd sees your peer(s), then it should be bringing the system back into sync. Looks like everything is good now, though. I just hoped to provide some extra insight. Hope this helps in the future! – Aaron Copley Jan 10 '11 at 21:00
  • I was using ntpq to check if ntp was doing its job -- IE, to see if the system clock is off or not. – John Bachir Jan 18 '11 at 15:46
  • The `-b` option should be used with ntpd to set the clock on boot without slowly slewing the clock. This option is commonly used on devices that don't have a battery backed RTC. Also, you really should have several sources for time, not just rely on a single server. – Chris S Jan 18 '11 at 16:08

9 Answers9

12

You can enable logging in ntpd by adding this to ntp.conf:

logfile /var/log/ntpd.log

Source: ntp manual

If you turn off ntpd, can you update the clock by command line? If you run the ntpdate command and receive an error like so:

# ntpdate ntp.ubuntu.com
10 Jan 23:47:57 ntpdate[26284]: Can't adjust the time of day: Operation not permitted

This means that you are probably on a VPS, and in that case you can not modify the system clock - this can only be done on the host machine.

Dave Drager
  • 8,375
  • 29
  • 45
  • ntpdate is happy to do its thing -- But I wonder if when my server reboots the clock gets reset to the hardware's clock or some such thing? – John Bachir Jan 10 '11 at 20:53
  • After you use ntpdate to set the clock, use 'hwclock --systohc' to sync "running" time to your hardware clock. It is supposed to sync on a reboot but if your machine crashes (or otherwise had a problem doing a proper shutdown) it could not sync it. – Dave Drager Jan 10 '11 at 20:54
  • Well it's a vhost, so I don't have access to the hardware clock (at least I hope not!) – John Bachir Jan 10 '11 at 21:05
  • There's an emulated hardware clock just as there is emulated BIOS. – Keith Stokes Jan 10 '11 at 22:27
  • I was the admin on several VPS platforms, none of them (openvz, Xen) have access to set the clock on the system. They all had to be done at the host level. Submit a ticket with your host to indicate the time is off, they should be running ntp and have the time synchronized for you. – Dave Drager Jan 11 '11 at 02:47
  • Is it possible that `sudo ntpdate ntp.ubuntu.com` succeeds, but isn't "really" working? Or does the failure of that command correspond 1-1 with my inability to set the clock in a VPS? – John Bachir Jan 17 '11 at 21:03
  • You might also need to use sudo... – Cerin Sep 13 '11 at 13:53
7

Alright folks, in the time since asking this question, I've reinstalled ntp with the default vendor (Ubuntu 10.0.4) config and let it run for a few days. As of this writing, ntpdate -q ntp.ubuntu.com shows that my time is accurate to within 0.000216 seconds. So, the problems I was having must have been with my customized config (where I was trying to make it impossible for external hosts to query my server, which I'm already doing with my firewall so I'm not too worried about). Here is the Ubuntu 10.0.4 ntp.conf in its entirety, with comments removed:

driftfile /var/lib/ntp/ntp.drift

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

server ntp.ubuntu.com

restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

restrict 127.0.0.1
restrict ::1

I welcome feedback on how this config might be improved.

I also made a ticket with my VPS provider asking them for a detailed recommendation on the best thing to do. I pointed them to this thread, and some other documentation indicating that maybe the CPU allocation would cause a timing problem. Here is what they said:

The latest kernels should not lock your system to our dom0's clock, to be on the safe side you can set xen.independent_wallclock = 1 in your sysctl.conf. This will make sure the server instance isn't following the clock on the host server.

and:

I think you may be mis-understanding the exact degree to which this issue affects NTP clients in a virtualized environment. In my experience on a virtualized system on a Xen host (such as our setup at Rackspace Cloud) the inaccuracy inherited by not having a dedicated system clock to process the interrupts amounts to fractions of a second, even on highly loaded systems. This slight inaccuracy is easily managed by NTP even if it is only set to update the servers time once per day (or even less frequent than that).

John Bachir
  • 2,364
  • 7
  • 29
  • 37
4

One of your comments says you are running on a vhost. In this case, you are probably not going to have much success because your vhost's sense of time is going to be dependent both on the real host it is running on and by how busy overall the vhost is.

Depending on the virtualization used, the vhost may not get a steady share of interrupts in a given time period. This will make the clock run faster, or slower, than is really happening. Since ntp is trying to measure changes on the assumption that your clock is a fixed-rate faster or slower than the rest of the world, this speeding up and slowing down will give ntp fits and it will probably eventually just give up, with the result that ntp -np shows time servers that ntp has deemed unsuitable.

Your best bet if this is the case is probably a brute-force rdate -s $server every so often (like every six hours) to yank the clock around by its nose so that it doesn't drift excessively out of sync. But fine-grained accuracy is probably out of reach.

David Mackintosh
  • 14,293
  • 7
  • 49
  • 78
4

Things I've found in the past, when I used ntpd instead of openntpd:

  1. You need to permit access to localhost for ntpd to start properly and actually do stuff

    restrict 127.0.0.1
restrict ::1

  2. Although you can use hostnames for server rules, opening back up holes to talk with those servers means using restrict which requires IP addresses, so I ended up having to use IPs for everything anyway.

  3. You don't mention using restrict to open access back up to your servers. That's an issue. Try blocks such as the following:

    # ntp.xs4all.nl
server          194.109.22.18
restrict        194.109.22.18

  4. You need multiple peers or servers for ntpd, since it tries to use majority-rules voting to deal with a bad actor. So a minimum of 4, to still be able to have a majority when you lose one, preferably 5.

  5. To lock down the default access, I could use:

    restrict default notrust nomodify

    so as to still be able to query, but I ended up using restrict default ignore as you do when ntpd 4.2 changed the meaning of notrust. sigh

  6. If you're not providing time-service to others, then you probably don't need the full power of regular ntpd and you should consider openntpd instead. Written by the OpenBSD crew, it's a far more minimal implementation, using privilege separation and a much simpler config file. It allegedly won't provide the highly precise time that ntpd will, but it's easily good enough for a regular server or workstation.

Phil P
  • 3,080
  • 1
  • 16
  • 19
  • This is great info. I'm checking out openntpd. Question: do you concur or disagree with the other folks who claim that setting the clock on a vhost is impossible? – John Bachir Jan 17 '11 at 20:30
  • And perhaps you can answer this question: http://serverfault.com/questions/223511/how-often-does-openntpd-update-the-time – John Bachir Jan 17 '11 at 20:48
  • I don't understand what you are saying with your various `restrict` rules… do those rules affect which servers I can query for time too? I thought it only affected which nodes can ask me for time. – John Bachir Jan 17 '11 at 21:09
  • 1
    Here's an idea: wanna change your answer into a complete minimal ntp.conf file with comments? :-) – John Bachir Jan 17 '11 at 21:12
  • Setting the clock on a vhost is ill-advised, unless you're guaranteed to always be scheduled with at least one CPU, since otherwise the time perceived by the vhost does not match the time of the outside world. The dom0 should be maintaining time. The other question's answer is good. NTP is UDP, so you need to permit packets from the servers which you query for time. My ntpd is stale as I moved to OpenNTPD a few years ago. – Phil P Jan 18 '11 at 00:27
  • Phil P: Thanks for all your info and ideas. See my submitted and accepted answer for my resolution to this, and let me know what you think of that config. – John Bachir Jan 20 '11 at 20:16
4
  • If ntpd wouldn't be able to connect with the remote server, you wouldn't see an offset for that server.
  • If ntpq would be blocked by ntpd, you would see a clear error message from ntpq.
  • If some other service would set also the time (like vmware tools), you would see a jumping offset for the server (run ntpq -p every 70 seconds).

The reach 7 in ntpq output indicated, that you let ntpd only run for around 4 minutes. 7 is 111 binary, which means the server was reached already 3 times. ntp reaches out every 64 seconds (poll value) and waited already 30 seconds (when value) since the last contact.

The offset -0.136 indicated, that the system is already synchronized. Only ntpd hasn't marked the server as source, yet. Just give it more time and a little star will appear.

So, actually your ntpd was syncing. But ntpd doesn't usually sync in one big jump (like ntpdate), but tries to adjust the time slowly and ensures over several cycles, that the time is stable.

PS: I am aware, that the question is very old. But the issue is timeless. And all the other answers are just misleading IMHO. ntpd is even recommended by VMWare to keep the time in sync.

Robert
  • 41
  • 1
2

I found my system off and puzzled why the HW clock wasn't sync'ing with the system clock on a clean shutdown. Seems there's an NTP setting in sysconfig that needs editing to make that happen.

In /etc/sysconfig/ntpd:

# Set to 'yes' to sync hw clock after successful ntpdate
SYNC_HWCLOCK=no

I set that to yes. Of course first verify that you have a solid NTP server and your system clock is reliable.

I knew that was it - my skew was 47 seconds and my HW clock was also 47 seconds off. Bingo! My first clue was Kerberos failures seen in the logs. Kerberos and many NAS will just not work if clock skew is too great.

Have a Nice Day!

jscott
  • 24,484
  • 8
  • 79
  • 100
Wayne Sweatt
  • 21
  • 2
1

This was not OP’s issue, but another reason for ntpd not syncing could be that the ntp port (123) is blocked (e.g. by the company’s firewall).

A solution is to use htpdate, which has a daemon mode (-D).

[credits]

Skippy le Grand Gourou
  • 1,205
  • 17
  • 24
0

if you are running vhost on vmware check the following article..it shuld help you http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf

Elgreco08
  • 228
  • 3
  • 4
  • 16
0

Hai..

Take a look at this reference to see if it can assist in troubleshooting your issue:

http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:Ch24:_The_NTP_Server

you might want to post the contents of your ntpd.conf file, the output of the debug commands like ntpq -p

And check your date/time?

And verify this also, run ntpdate and startup ntpd, is the time keeping in sync?

with best wishes

Premjith
  • 82
  • 1
  • 2
  • 11