I thought my watch had stopped. Or was running wild. It was happening again. Just a few months after I cracked the case of the slipped time reference site in the case I called "Pssst, ABAP Detective, you got a second?", I found systems with differing times. My first clue was a message saying there were 7 seconds missing from a job. This time (no pun intended), the fault wasn't directly seen in an SAP system. I started digging further.
In the last time drift episode, I found clues inside of SAP job logs that led me to review what sources of time synchronization were defined. I couldn't just ask the railroad conductor where he got his timetable from before he blew the train whistle, I needed to discover it for myself. The first pattern was Windows systems having one time, yet UNIX systems having another. When I first checked, it was seven seconds; when I looked later, it was up to 10 seconds and climbing. Not a lot in the grand scheme of things, but if some timestamp got whacky the shippers could lose packages, or something.
Here were the lessons from the last drift:
- Don't use just 2 reference sites. Use 3.
- Use the NTP pools as reference.
- Make sure your reference sites exist.
I could sense these didn't solve everything. To keep this tale short (and maybe sweet), I found out the Windows systems get their time from Active Directory domain controllers. That's fine, if those controllers are straight. If they're crooked, well, the whole continent was drifting out to sea with them, climate change or not. My first new clue was to use a little-known Window utility, since "ntpdate" isn't part of the standard OS.
Rap sheet on remote reference system
$ w32tm /stripchart /computer:0.us.pool.ntp.org /samples:5 /dataonly Tracking 0.us.pool.ntp.org [64.95.243.61]. Collecting 5 samples. The current time is 11/13/2012 4:50:01 PM (local time). 16:50:01, -10.5314154s 16:50:03, -10.5518846s 16:50:05, -10.5496261s 16:50:07, -10.5502981s 16:50:09, -10.5501962s |
Rap sheet on domain controller
$ w32tm /stripchart /computer:domcol /samples:5 /dataonly Tracking domcol [1.2.3.4]. Collecting 5 samples. The current time is 11/13/2012 4:47:48 PM (local time). 16:47:48, -00.7965628s 16:47:50, -00.7883558s 16:47:52, -00.7879628s 16:47:54, -00.7953808s 16:47:56, -00.7793569s |
- MS information on w32tm. "A tool used to diagnose problems occurring with Windows Time"
To make sure my head was on straight, my pocket watch was intact, and I wasn't looking at things in a mirror, I also found a distribution of NTP client for Windows, which included the same "ntpdate" command I had used to run down the culprits in the last chase.
C:\Program Files\NTP\bin>.\ntpdate.exe -d 2.north-america.pool.ntp.org 15 Nov 02:40:24 ntpdate.exe[8040]: ntpdate 4.2.6p5@1.2349-o Jul 30 11:53:32 (UTC +02:00) 2012 (1) 15 Nov 02:40:24 ntpdate.exe[8040]: Raised to realtime priority class transmit(50.115.174.206) receive(50.115.174.206) transmit(64.73.32.135) transmit(69.50.219.51) receive(64.73.32.135) [...] receive(207.5.137.133) server 50.115.174.206, port 123 stratum 2, precision -21, leap 00, trust 000 refid [50.115.174.206], delay 0.10367, dispersion 0.00319 transmitted 4, in filter 4 reference time: d44ed3b3.36942e53 Thu, Nov 15 2012 2:36:35.213 originate timestamp: d44ed493.c8d76175 Thu, Nov 15 2012 2:40:19.784 transmit timestamp: d44ed49f.329708c5 Thu, Nov 15 2012 2:40:31.197 filter delay: 0.15057 0.11934 0.10367 0.11923 0.00000 0.00000 0.00000 0.00000 filter offset: -11.4821 -11.4584 -11.4598 -11.4600 0.000000 0.000000 0.000000 0.000000 delay 0.10367, dispersion 0.00319 offset -11.459886 |
Pretty much the same result, though now, a day or so later, the delta is over 11 seconds. I'm sure some of you are saying what difference does a few seconds make? In my world, a lot. It's not keeping me awake at night, but it's my mission to set the servers straight. And if their controllers are crooked, well, we can vote them out of office, or we can take them downtown.
References
- NTP Client For Windows ("NT"): http://www.meinberg.de/english/sw/ntp.htm
- General NTP (Network Time Protocol) Server and Client information: http://www.eecis.udel.edu/~mills/ntp
- NTP and Windows details: http://www.eecis.udel.edu/~mills/ntp/html/hints/winnt.html
- "Time synchronization may not succeed when you try to synchronize with a non-Windows NTP server in Windows Server 2003" Microsoft Knowledge base: 875424
Crude purple crayon drawings
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
- SAP Managed Tags:
8 Comments
