Leap second - 2015 Study

What is it

In order to sync UTC (Coordinated Universal Time) with TAI (International Atomic Time), the day of 30th June 2015 will have 86401s instead of 86400s.

When does it come

Starting from 1972 there have been several leap seconds set. Last one was on 30th June 2012.

Year	Jun 30	Dec 31
1972	+1	+1
...	...	...
2008	0	+1
2012	+1	0
2015	+1

Clock Disciplines

Obviously the time syncing information is received through NTP but the system's clock is disciplined by either Kernel of by the NTP itself.

NTP discipline

Normally, the time is slewed if the offset is less than the step threshold, which is 128 ms by default, and stepped if above the threshold. The '-x' option for ntpd that will be discussed below, sets the threshold to 600 s. Since the slew rate of typical kernels is limited to 0.5 ms/s, each second of adjustment requires an amortization interval of 2000 s.   If '-x' option is set the kernel time discipline is disabled.

Kernel discipline

If the NTP program runs on a Linux version which supports kernel discipline (all current kernels supports), then the NTP client-daemon just passes a leap second down to the kernel which handles the leap second. The way that the kernel discipline the clock at leap second insertion is just stepping one second.

How does Linux get to know

There are three methods that Linux well get aware about the leap  second and do the synchronization.

NTP announcement

During the last day before a leap second correction, NTP servers should notify their clients that a leap second will occur. Whether to follow NTP clock discipline or Kernel clock discipline is depend on the NTP client configuration.
The leap second announcement packet content is given below. Note that the 'Leap indicator: +1s'.

Content of the packet using tcpdump

Server, Leap indicator: +1s (64), Stratum 3, poll 8s, precision -6     Root Delay: 0.188217, Root dispersion: 0.117919, Reference-ID: 10.124.124.20       Reference Timestamp:  3644696646.420012712 (2015/06/30 23:44:06)       Originator Timestamp: 3639291973.586123943 (2015/04/29 10:26:13)       Receive Timestamp:    3644697550.648012697 (2015/06/30 23:59:10)       Transmit Timestamp:   3644697550.648012697 (2015/06/30 23:59:10)         Originator - Receive Timestamp:  +5405577.061888758         Originator - Transmit Timestamp: +5405577.061888758

ZONE file update

For RHEL, it is also possible to make the reporting time corrected for leap seconds by updating the tzdata package to the latest version available, copying the appropriate file from the /usr/share/zoneinfo/right directory hierarchy to /etc/localtime, and resetting the clock to the correct local time.  The files in /usr/share/zoneinfo/right contain local time information corrected for all leap seconds that have occurred since the beginning of the Epoch on 1970-01-01 00:00:00 UTC.

Example for Colombo

# yum update tzdate # ln -s /usr/share/zoneinfo/right/Asia/Colombo /etc/localtime

Manually set/unset

It is also possible to trigger leap second using the Kernel clock discipline by using adjtimex() system call. In SLES there is a tool called 'adjtimex' for RHEL we may have to write a program. The Kernel time variable should be set on the day before leap second and it should be disarmed next day so it can be scheduled using cronie.

Crontab entry for manual leap second

59 23 30 6 * root 1 0 1 7 * root

With ntptime (for SLES and RHEL)

Set: # ntptime -s 16 Unset: # ntptime -s 0

SLES

Set leap second flag

Set: # adjtimex -s16 Unset: # adjtimex -s0

RHEL

example code to set leap second

#include                                                           #include                                                               #include                                                              int main(){    struct timex tx;    tx.modes=ADJ_STATUS;                                                               tx.status=STA_INS;                                                                   if(adjtimex(&tx) == -1)                                                            {                                                                                     perror("adjtimex(1)");                                                               exit(1);                                                                          }    printf("Set!");                                                           } Compile and run # ./ls Set!

How to identify if the server is set for leap second

When using NTP discipline

Check if leap announce is recieved

# ntpq -c rl | grep 'leap=' processor="x86_64", system="Linux/3.0.74-0.6.10-default", leap=01,

'leap' Value	Meaning
00	no warning
01	last minute of the day has 61 seconds
10	last minute of the day has 59 seconds
11	unknown (clock unsynchronized)

When using Kernel discipline

With ntptime command in both SLES and RHEL. If the status contains INS flag that means a leap second insertion is set.

# ntptime | grep status   status 0x50 (INS,UNSYNC),

SLES

adjtimex -p print the current values of the kernel time variables.  "status" gives the value of the time_status variable in the kernel. The value is a sum of the following

1	PLL updates enabled
2	PPS freq discipline enabled
4	PPS time discipline enabled
8	frequency-lock mode enabled
16	inserting leap second
32	deleting leap second
64	clock unsynchronized
128	holding frequency
256	PPS signal present
512	PPS signal jitter exceeded
1024	PPS signal wander exceeded
2048	PPS signal calibration error
4096	clock hardware fault

# adjtimex -p          mode: 0        offset: 0     frequency: 0      maxerror: 16000000      esterror: 16000000        status: 16 time_constant: 2     precision: 1     tolerance: 32768000          tick: 10000      raw time:  1431048543s 786257us = 1431048543.786257  return value = 5

RHEL

print leap status

#include                                                           #include                                                               #include                                                              int main(){                                                                                  struct timex tx;                                                                   tx.modes=0;                                                                        if(adjtimex(&tx) == -1){                                                                                     perror("adjtimex(2)");                                                               exit(1);                                                                          }                                                                                  printf("Kernel leap second flag: ");                                               if(tx.status & STA_INS)                                                               printf("add\n");                                                                  else                                                                                  if(tx.status & STA_DEL)                                                                 printf("delete\n");                                                                 else                                                                                    printf("not set\n");                                                          }

Command output

Output when not set $ ./lg Kernel leap second flag: not set Output when set to add 1s $ ./lg Kernel leap second flag: add

When the leap second is applied with Kernel discipline you will find following message in dmesg

leap insertion message in dmesg

# dmesg | grep -i leap [  648.669570] Clock: inserting leap second 23:59:60 UTC

How to Setup a Demo Lab

Setup a NTP server with leap second announcement

1. Install the ntp server
   SLES: zypper in ntp
   RHEL: yum install ntp
2. Get the leap file from NIST.
   cd /var/ntp
   sudo wget http://www.ietf.org/timezones/data/leap-seconds.list
3. Then we need to change settings of ntpd so that it can announce a fake time.
   sudo vi /etc/ntp.conf
4. Comment all the `server` lines out, and set the server itself as refclock
   server 127.127.1.0
   fudge 127.127.1.0 stratum 10
5. Set the leap file
   leapfile "/var/ntp/leap-seconds.list" 
6. The resulting /etc/ntp.conf file will look like this
   

   # grep -v ^# /etc/ntp.conf server 127.127.1.0 fudge  127.127.1.0 stratum 10 leapfile /var/ntp/leap-seconds.list driftfile /var/lib/ntp/drift/ntp.drift keys /etc/ntp.keys      trustedkey 1 requestkey 1

7. Now we set the system time to somewhere a little before a leap second took place, e.g, Jun 30, 2015, and then restart the ntpd daemon.
   export TZ=UTC
   date -s "2015-06-30 23:50"
   service ntp restart
8. Now we just got back to 10 minutes earlier than a leap-second in the past.

Setup a NTP client with statistics enabled

1. Install the ntp server
   SLES: zypper in ntp
   RHEL: yum install ntp 
2. Change the settings of the ntp to use above server.
   sudo vi /etc/ntp.conf
3. Add ntp server
   server
4. Create directory for statistics
   mkdir /var/lib/ntp/tmp/
5. Enable statistics collection - uncomment following lines or add if not available. Note that ntp root is /var/lib/ntp so /tmp/ is equivalent to /var/lib/ntp/tmp/
   statsdir /tmp/        # directory for statistics files
   filegen peerstats  file peerstats  type day enable
   filegen loopstats  file loopstats  type day enable
   filegen clockstats file clockstats type day enable
6. The resulting ntp.conf file will look like this
   

   server 172.25.37.20 driftfile /var/lib/ntp/drift/ntp.drift      logfile /var/log/ntp             statsdir /tmp/        # directory for statistics files filegen peerstats  file peerstats  type day enable filegen loopstats  file loopstats  type day enable filegen clockstats file clockstats type day enable keys /etc/ntp.keys             trustedkey 1                 requestkey 1

7. Now sync the time with the ntp server using one-time-mode.
   # ntpdate -q
8. Now start ntpd daemon
   # service ntp start

NTP configuration with NTP discipline

By default ntpd will run with Kernel discipline mode so when it receive the leap second anouncement it will arm kernel with it. If you want to use NTP discipline mode ntpd should start with '-x' option.

SLES

vim /etc/sysconfig/ntp
NTPD_OPTIONS=" -x -g -u ntp:ntp"

RHEL

vim /etc/sysconfig/ntpd
OPTIONS=" -x -u ntp:ntp -p /var/run/ntpd.pid -g"

Setup a NTP proxy with leap second announcement

NTP server with a drifting clock like the one in the server will provide drifting sync for the client. So setting a ntp server may cause complications to the testing. There is a ntp proxy wrote by Robert Karbowski which is available in github. Download it and make, so you will have two binaries 'ntpproxy' and 'sls'.

• ntpproxy will add an offset to time of the source ntp server that is somewhere close to the UTC 23:59:59 of 30th June 2015 (default is 600s before leap second) and forward to the client. Clients queries will also transformed accordingly and forward to the server.
• It will also do the leap announcement to the clients.
• ntp client configuration will be similar to the one given above.

starting ntp proxy

# ./ntpproxy -s -v example using internal ntp server: # ./ntpproxy -s 172.25.123.20 -v

Manual

Robert Karbowski also provides a binary called 'sls' we can use this to manually set the kernel time variables for leap second.

• sls will also set the system time close to the UTC 23:59:59 of 30th June 2015 (default is 600s before leap second)
  

We can also use adjtimex to set the kernel time variable to demonstrate the leap second.

# export TZ=UTC # date -s "`date +%Y%m-%d` 23:59:00" # adjtimex -S16

Check time during test

Following code (clockm) will print the time in 1ms intervals and the printing time format and the functions given below.
Formate:

gettimeofday	clock_gettime(CLOCK_MONOTONIC
::	::
23:59:50   31	00:14:08  526

clockm

#include #include #include int main(void) {   char buffer[30],buffer_[30];   struct timeval  tv1;   time_t curtime,mtime;   struct timespec ts1;   while(1){      gettimeofday(&tv1, NULL);      clock_gettime(CLOCK_MONOTONIC,&ts1);      curtime=tv1.tv_sec;      strftime(buffer,30,"%T ",localtime(&curtime));      printf("%s %ld; ",buffer,tv1.tv_usec/1000);      curtime=ts1.tv_sec;      strftime(buffer,30,"%T ",localtime(&curtime));      printf("%s %ld\n",buffer,ts1.tv_nsec/1000000);      usleep(1000);   }   return 0; }

Tests

Test  the Kernel discipline with clockm

Following script was used to test the kernel discipline when the leap second.

leapset.sh

#!/bin/bash export TZ=UTC adjtimex -S0 date -s "`date +%Y-%m-%d` 23:59:50" #date -s "2015-06-30 23:59:50" adjtimex -S16 ./clockm | tee -a leap-$$.log

Observations

OS	Kernel	Clock Discipline	Observations	Comments
SLES 11 SP3	3.0.101-0.47.52-default	Kernel	... 23:59:59  998; 00:02:10  44 23:59:59  999; 00:02:10  45 00:00:00  0; 00:02:10  46 00:00:00  1; 00:02:10  47 23:59:59  3; 00:02:10  49 23:59:59  4; 00:02:10  50 ... 23:59:59  997; 00:02:11  43 23:59:59  998; 00:02:11  44 23:59:59  999; 00:02:11  45 00:00:00  0; 00:02:11  46 00:00:00  1; 00:02:11  47 00:00:00  2; 00:02:11  48 ...	There were two 59th seconds observed 00th second reported between first 59th second and  second 59th second (while switching).
SLES11 SP2	3.0.13-0.27-default	Kernel	... 23:59:59  996; 01:24:29  377 23:59:59  998; 01:24:29  379 23:59:59  999; 01:24:29  380 00:00:00  0; 01:24:29  381 00:00:00  1; 01:24:29  382 00:00:00  2; 01:24:29  383 ...
RHEL 6.2	2.6.32-431.5.1.el6.x86_64	Kernel	... 23:59:59  997; 00:18:02  325 23:59:59  998; 00:18:02  326 23:59:59  999; 00:18:02  327 23:59:59  1; 00:18:02  329 23:59:59  2; 00:18:02  330 23:59:59  3; 00:18:02  331 ...
RHEL 6.2	2.6.32.220.el6.x86_64		...  23:59:59  996; 00:04:10  177  23:59:59  998; 00:04:10  178  23:59:59  999; 00:04:10  179  23:59:59  0; 00:04:10  180  23:59:59  1; 00:04:10  181  23:59:59  2; 00:04:10  182 ...	Elapsed two seconds for 59

Test the NTP discipline with clockm

Observations

Check if the installed kernel and package versions in RHEL are vulnerable

RedHat has provided a script to perform this called leap_vulnerability.sh.
If the target is vulnerable, you will see output similar to:

$ ./leap_vulnerability.sh This system is vulnerable to a performance degradation after the Leap Second Insertion of June 30, 2015. Please refer to //access.redhat.com/solutions/154793> for remediation steps.

If the target is not vulnerable, you will see output similar to:

$ ./leap_vulnerability.sh Not vulnerable

Test with leap-a-day

RedHat has provided a code leap-a-day.c, for checking a system for 'known hrtimer failure' while leap second was applied.

OS	Kernel	Clock Discipline	Observations	Comments
SLES11SP2	3.0.13-0.27	Kernel	root@vm_perl_dev|SLES11SP2|3.0.13-0.27:~ # ./leap-a-day -s Setting time to speed up testing This runs continuously. Press ctrl-c to stop Setting time to Thu May 14 23:59:50 2015 Scheduling leap second for Fri May 15 00:00:00 2015 Thu May 14 23:59:57 2015 + 195 us (0) TIME_INS Thu May 14 23:59:57 2015 + 500303 us (0) TIME_INS Thu May 14 23:59:58 2015 + 431 us (0) TIME_INS Thu May 14 23:59:58 2015 + 500546 us (0) TIME_INS Thu May 14 23:59:59 2015 + 682 us (0) TIME_INS Thu May 14 23:59:59 2015 + 500853 us (0) TIME_INS Thu May 14 23:59:59 2015 + 1049 us (0) TIME_OOP Thu May 14 23:59:59 2015 + 501154 us (0) TIME_OOP Fri May 15 00:00:00 2015 + 1268 us (0) TIME_WAIT Fri May 15 00:00:00 2015 + 501398 us (0) TIME_WAIT Fri May 15 00:00:01 2015 + 1556 us (0) TIME_WAIT Fri May 15 00:00:01 2015 + 501687 us (0) TIME_WAIT Fri May 15 00:00:02 2015 + 1847 us (0) TIME_WAIT ERROR: hrtimer early expiration failure observed. Leap complete Setting time to Fri May 15 23:59:50 2015 Scheduling leap second for Sat May 16 00:00:00 2015 Fri May 15 23:59:57 2015 + 51 us (0) TIME_DEL Fri May 15 23:59:57 2015 + 500199 us (0) TIME_DEL Fri May 15 23:59:58 2015 + 352 us (0) TIME_DEL Fri May 15 23:59:58 2015 + 500527 us (0) TIME_DEL Sat May 16 00:00:00 2015 + 634 us (0) TIME_WAIT Sat May 16 00:00:00 2015 + 500781 us (0) TIME_WAIT Sat May 16 00:00:01 2015 + 954 us (0) TIME_WAIT Sat May 16 00:00:01 2015 + 501123 us (0) TIME_WAIT Sat May 16 00:00:02 2015 + 1332 us (0) TIME_WAIT Leap complete
SLES11SP3	3.0.101-0.47.52	Kernel	OK
SLES11SP2	3.0.101-0.7.27	Kernel	OK
SLES11SP1	2.6.32.29-0.3	Kernel	ERROR: hrtimer early expiration failure observed.

Recommendations

NTP

SUSE	Run ntp in slew mode	Stop ntpd # rcntp stop 2. Reset the kernel status and frequency # ntptime -s 0 -f 0  3. Configure ntp with the -x option (ntp slew mode) Edit /etc/sysconfig/ntp and add -x option as follows NTPD_OPTIONS=" -x -g -u ntp:ntp" 4. Start ntpd # rcntp start
Red Hat	Run ntp in slew mode	Update ntp and ntpdate to the following versions: ntpdate-4.2.6p5-3.el6_6.x86_64 ntp-4.2.6p5-3.el6_6.x86_64 2. Stop ntpd # service ntpd stop 3. Reset the kernel status and frequency # ntptime -s 0 -f 0  4. Configure ntp with the -x option (ntp slew mode) Edit /etc/sysconfig/ntpdOPTIONS=" -x -u ntp:ntp -p /var/run/ntpd.pid -g" 5. Start ntpd # service ntpd start
Solaris	Run ntp in slew mode	Stop NTP service 24 hours prior to the leap second. # svcadm disable ntp  2. Add following 2 lines in the /etc/inet/ntp.conf slewalways yes disable pll 3. Start ntp # svcadm enable ntp
OS	Description of the Solution	What Does the Client Have To Do?

References

https://github.com/AmadeusITGroup/NTP-Proxy/blob/master/README.md
https://www.meinberg.de/download/burnicki/Technical%20Aspects%20of%20Leap%20Second%20Propagation%20and%20Evaluation.pdf
https://www.suse.com/support/kb/doc.php?id=7016150
https://access.redhat.com/articles/15145
https://www.suse.com/support/kb/doc.php?id=7016355
https://access.redhat.com/articles/199563
http://license.fsmlabs.com/timekeeper_release/timekeeper.pdf
https://support.microsoft.com/en-us/kb/909614
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=427811617623679&id=1019692.1&_afrWindowMode=0&_adf.ctrl-state=6cqxkfi1h_4