Wednesday, June 24, 2015

Leap second - 2015 Study

What is it

In order to sync UTC (Coordinated Universal Time) with TAI (International Atomic Time), the day of 30th June 2015 will have 86401s instead of 86400s.

When does it come

Starting from 1972 there have been several leap seconds set. Last one was on 30th June 2012.
Year
Jun 30
Dec 31
1972+1+1
.........
20080+1
2012+10
2015+1

Clock Disciplines

Obviously the time syncing information is received through NTP but the system's clock is disciplined by either Kernel of by the NTP itself.

NTP discipline

Normally, the time is slewed if the offset is less than the step threshold, which is 128 ms by default, and stepped if above the threshold. The '-x' option for ntpd that will be discussed below, sets the threshold to 600 s. Since the slew rate of typical kernels is limited to 0.5 ms/s, each second of adjustment requires an amortization interval of 2000 s.   If '-x' option is set the kernel time discipline is disabled.

Kernel discipline

If the NTP program runs on a Linux version which supports kernel discipline (all current kernels supports), then the NTP client-daemon just passes a leap second down to the kernel which handles the leap second. The way that the kernel discipline the clock at leap second insertion is just stepping one second.

How does Linux get to know

There are three methods that Linux well get aware about the leap  second and do the synchronization.

NTP announcement

During the last day before a leap second correction, NTP servers should notify their clients that a leap second will occur. Whether to follow NTP clock discipline or Kernel clock discipline is depend on the NTP client configuration.
The leap second announcement packet content is given below. Note that the 'Leap indicator: +1s'.
Content of the packet using tcpdump
Server, Leap indicator: +1s (64), Stratum 3, poll 8s, precision -6
    Root Delay: 0.188217, Root dispersion: 0.117919, Reference-ID: 10.124.124.20
      Reference Timestamp:  3644696646.420012712 (2015/06/30 23:44:06)
      Originator Timestamp: 3639291973.586123943 (2015/04/29 10:26:13)
      Receive Timestamp:    3644697550.648012697 (2015/06/30 23:59:10)
      Transmit Timestamp:   3644697550.648012697 (2015/06/30 23:59:10)
        Originator - Receive Timestamp:  +5405577.061888758
        Originator - Transmit Timestamp: +5405577.061888758

ZONE file update

For RHEL, it is also possible to make the reporting time corrected for leap seconds by updating the tzdata package to the latest version available, copying the appropriate file from the /usr/share/zoneinfo/right directory hierarchy to /etc/localtime, and resetting the clock to the correct local time.  The files in /usr/share/zoneinfo/right contain local time information corrected for all leap seconds that have occurred since the beginning of the Epoch on 1970-01-01 00:00:00 UTC.
Example for Colombo
# yum update tzdate
# ln -s /usr/share/zoneinfo/right/Asia/Colombo /etc/localtime

Manually set/unset

It is also possible to trigger leap second using the Kernel clock discipline by using adjtimex() system call. In SLES there is a tool called 'adjtimex' for RHEL we may have to write a program. The Kernel time variable should be set on the day before leap second and it should be disarmed next day so it can be scheduled using cronie.
Crontab entry for manual leap second
59 23 30 6 * root
1 0 1 7 * root

With ntptime (for SLES and RHEL)

Set:
# ntptime -s 16
Unset:
# ntptime -s 0

SLES

Set leap second flag
Set:
# adjtimex -s16
Unset:
# adjtimex -s0

RHEL

example code to set leap second
#include                                                          
#include                                                              
#include                                                             
int main(){
   struct timex tx;
   tx.modes=ADJ_STATUS;                                                           
   tx.status=STA_INS;                                                               
   if(adjtimex(&tx) == -1)                                                        
   {                                                                              
      perror("adjtimex(1)");                                                        
      exit(1);                                                                      
   }
   printf("Set!");                                                          
}
Compile and run
# ./ls
Set!

How to identify if the server is set for leap second

When using NTP discipline


Check if leap announce is recieved
# ntpq -c rl | grep 'leap='
processor="x86_64", system="Linux/3.0.74-0.6.10-default", leap=01,
'leap' Value
Meaning
00no warning
01last minute of the day has 61 seconds
10last minute of the day has 59 seconds
11unknown (clock unsynchronized)

When using Kernel discipline

With ntptime command in both SLES and RHEL. If the status contains INS flag that means a leap second insertion is set.
# ntptime | grep status
  status 0x50 (INS,UNSYNC),

SLES

adjtimex -p print the current values of the kernel time variables.  "status" gives the value of the time_status variable in the kernel. The value is a sum of the following
1PLL updates enabled
2PPS freq discipline enabled
4PPS time discipline enabled
8frequency-lock mode enabled
16inserting leap second
32deleting leap second
64clock unsynchronized
128holding frequency
256PPS signal present
512PPS signal jitter exceeded
1024PPS signal wander exceeded
2048PPS signal calibration error
4096clock hardware fault

# adjtimex -p
         mode: 0
       offset: 0
    frequency: 0
     maxerror: 16000000
     esterror: 16000000
       status: 16
time_constant: 2
    precision: 1
    tolerance: 32768000
         tick: 10000
     raw time:  1431048543s 786257us = 1431048543.786257
 return value = 5

RHEL

print leap status
#include                                                          
#include                                                              
#include                                                             
int main(){                                                                              
   struct timex tx;                                                               
   tx.modes=0;                                                                    
   if(adjtimex(&tx) == -1){                                                                              
      perror("adjtimex(2)");                                                        
      exit(1);                                                                      
   }                                                                              
   printf("Kernel leap second flag: ");                                           
   if(tx.status & STA_INS)                                                        
      printf("add\n");                                                              
   else                                                                           
      if(tx.status & STA_DEL)                                                       
         printf("delete\n");                                                          
      else                                                                          
         printf("not set\n");                                                         
}
Command output
Output when not set
$ ./lg
Kernel leap second flag: not set
Output when set to add 1s
$ ./lg
Kernel leap second flag: add

When the leap second is applied with Kernel discipline you will find following message in dmesg
leap insertion message in dmesg
# dmesg | grep -i leap
648.669570] Clock: inserting leap second 23:59:60 UTC

How to Setup a Demo Lab

Setup a NTP server with leap second announcement

  1. Install the ntp server
    SLES: zypper in ntp
    RHEL: yum install ntp
  2. Get the leap file from NIST.
    cd /var/ntp
    sudo wget http://www.ietf.org/timezones/data/leap-seconds.list
  3. Then we need to change settings of ntpd so that it can announce a fake time.
    sudo vi /etc/ntp.conf
  4. Comment all the `server` lines out, and set the server itself as refclock
    server 127.127.1.0
    fudge 127.127.1.0 stratum 10
  5. Set the leap file
    leapfile "/var/ntp/leap-seconds.list" 
  6. The resulting /etc/ntp.conf file will look like this
    # grep -v ^# /etc/ntp.conf
    server 127.127.1.0
    fudge  127.127.1.0 stratum 10
    leapfile /var/ntp/leap-seconds.list
    driftfile /var/lib/ntp/drift/ntp.drift
    keys /etc/ntp.keys     
    trustedkey 1
    requestkey 1
  7. Now we set the system time to somewhere a little before a leap second took place, e.g, Jun 30, 2015, and then restart the ntpd daemon.
    export TZ=UTC
    date -s "2015-06-30 23:50"
    service ntp restart
  8. Now we just got back to 10 minutes earlier than a leap-second in the past.

Setup a NTP client with statistics enabled

  1. Install the ntp server
    SLES: zypper in ntp
    RHEL: yum install ntp 
  2. Change the settings of the ntp to use above server.
    sudo vi /etc/ntp.conf
  3. Add ntp server
    server
  4. Create directory for statistics
    mkdir /var/lib/ntp/tmp/
  5. Enable statistics collection - uncomment following lines or add if not available. Note that ntp root is /var/lib/ntp so /tmp/ is equivalent to /var/lib/ntp/tmp/
    statsdir /tmp/        # directory for statistics files
    filegen peerstats  file peerstats  type day enable
    filegen loopstats  file loopstats  type day enable
    filegen clockstats file clockstats type day enable
  6. The resulting ntp.conf file will look like this
    server 172.25.37.20
    driftfile /var/lib/ntp/drift/ntp.drift     
    logfile /var/log/ntp            
    statsdir /tmp/        # directory for statistics files
    filegen peerstats  file peerstats  type day enable
    filegen loopstats  file loopstats  type day enable
    filegen clockstats file clockstats type day enable
    keys /etc/ntp.keys            
    trustedkey 1                
    requestkey 1
  7. Now sync the time with the ntp server using one-time-mode.
    # ntpdate -q
  8. Now start ntpd daemon
    # service ntp start

NTP configuration with NTP discipline

By default ntpd will run with Kernel discipline mode so when it receive the leap second anouncement it will arm kernel with it. If you want to use NTP discipline mode ntpd should start with '-x' option.
SLES
vim /etc/sysconfig/ntp
NTPD_OPTIONS=" -x -g -u ntp:ntp"
RHEL
vim /etc/sysconfig/ntpd
OPTIONS=" -x -u ntp:ntp -p /var/run/ntpd.pid -g"

Setup a NTP proxy with leap second announcement

NTP server with a drifting clock like the one in the server will provide drifting sync for the client. So setting a ntp server may cause complications to the testing. There is a ntp proxy wrote by Robert Karbowski which is available in github. Download it and make, so you will have two binaries 'ntpproxy' and 'sls'.
  • ntpproxy will add an offset to time of the source ntp server that is somewhere close to the UTC 23:59:59 of 30th June 2015 (default is 600s before leap second) and forward to the client. Clients queries will also transformed accordingly and forward to the server.
  • It will also do the leap announcement to the clients.
  • ntp client configuration will be similar to the one given above.
starting ntp proxy
# ./ntpproxy -s -v
example using internal ntp server:
# ./ntpproxy -s 172.25.123.20 -v

Manual

Robert Karbowski also provides a binary called 'sls' we can use this to manually set the kernel time variables for leap second.
  • sls will also set the system time close to the UTC 23:59:59 of 30th June 2015 (default is 600s before leap second)

We can also use adjtimex to set the kernel time variable to demonstrate the leap second.
# export TZ=UTC
# date -s "`date +%Y%m-%d` 23:59:00"
# adjtimex -S16

Check time during test

Following code (clockm) will print the time in 1ms intervals and the printing time format and the functions given below.
Formate:
gettimeofday
clock_gettime(CLOCK_MONOTONIC
:: ::
23:59:50   3100:14:08  526
clockm
#include
#include
#include
int main(void)
{
  char buffer[30],buffer_[30];
  struct timeval  tv1;
  time_t curtime,mtime;
  struct timespec ts1;
  while(1){
     gettimeofday(&tv1, NULL);
     clock_gettime(CLOCK_MONOTONIC,&ts1);
     curtime=tv1.tv_sec;
     strftime(buffer,30,"%T ",localtime(&curtime));
     printf("%s %ld; ",buffer,tv1.tv_usec/1000);
     curtime=ts1.tv_sec;
     strftime(buffer,30,"%T ",localtime(&curtime));
     printf("%s %ld\n",buffer,ts1.tv_nsec/1000000);
     usleep(1000);
  }
  return 0;
}

Tests

Test  the Kernel discipline with clockm

Following script was used to test the kernel discipline when the leap second.
leapset.sh
#!/bin/bash
export TZ=UTC
adjtimex -S0
date -s "`date +%Y-%m-%d` 23:59:50"
#date -s "2015-06-30 23:59:50"
adjtimex -S16
./clockm | tee -a leap-$$.log

Observations

OS
Kernel
Clock Discipline
Observations
Comments
SLES 11 SP33.0.101-0.47.52-defaultKernel...
23:59:59  998; 00:02:10  44
23:59:59  999; 00:02:10  45
00:00:00  0; 00:02:10  46
00:00:00  1; 00:02:10  47
23:59:59  3; 00:02:10  49
23:59:59  4; 00:02:10  50
...
23:59:59  997; 00:02:11  43
23:59:59  998; 00:02:11  44
23:59:59  999; 00:02:11  45
00:00:00  0; 00:02:11  46
00:00:00  1; 00:02:11  47
00:00:00  2; 00:02:11  48
...
  • There were two 59th seconds observed
  • 00th second reported between first 59th second and  second 59th second (while switching).
SLES11 SP23.0.13-0.27-defaultKernel...
23:59:59  996; 01:24:29  377
23:59:59  998; 01:24:29  379
23:59:59  999; 01:24:29  380
00:00:00  0; 01:24:29  381
00:00:00  1; 01:24:29  382
00:00:00  2; 01:24:29  383
...
RHEL 6.22.6.32-431.5.1.el6.x86_64Kernel...
23:59:59  997; 00:18:02  325
23:59:59  998; 00:18:02  326
23:59:59  999; 00:18:02  327
23:59:59  1; 00:18:02  329
23:59:59  2; 00:18:02  330
23:59:59  3; 00:18:02  331
...
RHEL 6.22.6.32.220.el6.x86_64...
 23:59:59  996; 00:04:10  177
 23:59:59  998; 00:04:10  178
 23:59:59  999; 00:04:10  179
 23:59:59  0; 00:04:10  180
 23:59:59  1; 00:04:10  181
 23:59:59  2; 00:04:10  182
...
  • Elapsed two seconds for 59

Test the NTP discipline with clockm


Observations



Check if the installed kernel and package versions in RHEL are vulnerable

RedHat has provided a script to perform this called leap_vulnerability.sh.
If the target is vulnerable, you will see output similar to:
$ ./leap_vulnerability.sh
This system is vulnerable to a performance degradation after the Leap Second Insertion of June 30, 2015.
Please refer to //access.redhat.com/solutions/154793> for remediation steps.

If the target is not vulnerable, you will see output similar to:
$ ./leap_vulnerability.sh
Not vulnerable

Test with leap-a-day

RedHat has provided a code leap-a-day.c, for checking a system for 'known hrtimer failure' while leap second was applied.
OS
Kernel
Clock Discipline
Observations
Comments
SLES11SP2
3.0.13-0.27
Kernel
root@vm_perl_dev|SLES11SP2|3.0.13-0.27:~ # ./leap-a-day -s
Setting time to speed up testing
This runs continuously. Press ctrl-c to stop
Setting time to Thu May 14 23:59:50 2015
Scheduling leap second for Fri May 15 00:00:00 2015
Thu May 14 23:59:57 2015 +    195 us (0)    TIME_INS
Thu May 14 23:59:57 2015 + 500303 us (0)    TIME_INS
Thu May 14 23:59:58 2015 +    431 us (0)    TIME_INS
Thu May 14 23:59:58 2015 + 500546 us (0)    TIME_INS
Thu May 14 23:59:59 2015 +    682 us (0)    TIME_INS
Thu May 14 23:59:59 2015 + 500853 us (0)    TIME_INS
Thu May 14 23:59:59 2015 +   1049 us (0)    TIME_OOP
Thu May 14 23:59:59 2015 + 501154 us (0)    TIME_OOP
Fri May 15 00:00:00 2015 +   1268 us (0)    TIME_WAIT
Fri May 15 00:00:00 2015 + 501398 us (0)    TIME_WAIT
Fri May 15 00:00:01 2015 +   1556 us (0)    TIME_WAIT
Fri May 15 00:00:01 2015 + 501687 us (0)    TIME_WAIT
Fri May 15 00:00:02 2015 +   1847 us (0)    TIME_WAIT
ERROR: hrtimer early expiration failure observed.
Leap complete
Setting time to Fri May 15 23:59:50 2015
Scheduling leap second for Sat May 16 00:00:00 2015
Fri May 15 23:59:57 2015 +     51 us (0)    TIME_DEL
Fri May 15 23:59:57 2015 + 500199 us (0)    TIME_DEL
Fri May 15 23:59:58 2015 +    352 us (0)    TIME_DEL
Fri May 15 23:59:58 2015 + 500527 us (0)    TIME_DEL
Sat May 16 00:00:00 2015 +    634 us (0)    TIME_WAIT
Sat May 16 00:00:00 2015 + 500781 us (0)    TIME_WAIT
Sat May 16 00:00:01 2015 +    954 us (0)    TIME_WAIT
Sat May 16 00:00:01 2015 + 501123 us (0)    TIME_WAIT
Sat May 16 00:00:02 2015 +   1332 us (0)    TIME_WAIT
Leap complete
SLES11SP33.0.101-0.47.52Kernel
OK
SLES11SP23.0.101-0.7.27KernelOK
SLES11SP12.6.32.29-0.3Kernel
ERROR: hrtimer early expiration failure observed.


Recommendations

NTP

SUSERun ntp in slew mode
  1. Stop ntpd
# rcntp stop
2. Reset the kernel status and frequency
# ntptime -s 0 -f 0 
3. Configure ntp with the -x option (ntp slew mode)
Edit /etc/sysconfig/ntp and add -x option as follows
NTPD_OPTIONS=" -x -g -u ntp:ntp"
4. Start ntpd
# rcntp start
Red HatRun ntp in slew mode
  1. Update ntp and ntpdate to the following versions:
  • ntpdate-4.2.6p5-3.el6_6.x86_64
  • ntp-4.2.6p5-3.el6_6.x86_64
2. Stop ntpd
# service ntpd stop
3. Reset the kernel status and frequency
# ntptime -s 0 -f 0 
4. Configure ntp with the -x option (ntp slew mode)
Edit /etc/sysconfig/ntpdOPTIONS=" -x -u ntp:ntp -p /var/run/ntpd.pid -g"
5. Start ntpd
# service ntpd start
SolarisRun ntp in slew mode
  1. Stop NTP service 24 hours prior to the leap second.
# svcadm disable ntp
 2. Add following 2 lines in the /etc/inet/ntp.conf
slewalways yes
disable pll
3. Start ntp
# svcadm enable ntp
OS
Description of the Solution
What Does the Client Have To Do?

References

https://github.com/AmadeusITGroup/NTP-Proxy/blob/master/README.md
https://www.meinberg.de/download/burnicki/Technical%20Aspects%20of%20Leap%20Second%20Propagation%20and%20Evaluation.pdf
https://www.suse.com/support/kb/doc.php?id=7016150
https://access.redhat.com/articles/15145
https://www.suse.com/support/kb/doc.php?id=7016355
https://access.redhat.com/articles/199563
http://license.fsmlabs.com/timekeeper_release/timekeeper.pdf
https://support.microsoft.com/en-us/kb/909614
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=427811617623679&id=1019692.1&_afrWindowMode=0&_adf.ctrl-state=6cqxkfi1h_4

Intel Corporation XMM7360 LTE Advanced Modem in ArchLinux

The required packages are available in ArchLinux User Repository (AUR) so it will be easier if you can use an AUR helper package like yay . ...