Sunday, August 27, 2017

Find fastest mirror

This is for arch Linux but the simple technique can be used to sort any other mirrors.

First create a backup of the original /etc/pacman.d/mirrorlist as /etc/pacman.d/mirrorlist.orig

Then create the following simple script:

awk -F'/' '{print $3}' /etc/pacman.d/mirrorlist.orig | grep -v '^$' | while read line;
HEAD $line &>/dev/null && grep -B1 $line /etc/pacman.d/mirrorlist.orig&
done > /etc/pacman.d/mirrorlist

Running this script will create the sorted list of mirrors in /etc/pacman.d/mirrorlist based on how fast the mirror response for the HTTP request.

Monday, July 24, 2017

A simple thread manager for a bash/shell scripts

A simple thread manager for a bash/shell scripts. Place your command in place of 'sleep $((30/i))'

    :> $TREG
    function add_t(){
       echo Starting $1
       echo $1 >> $TREG
    function rem_t(){
       echo Done $1
       sed -i /^$1$/d $TREG
    function clean(){
       :> $TREG
       exit 1
    trap "clean" SIGINT SIGTERM
    for i in 1 2 3
       (add_t thread$i;sleep $((30/i));rem_t thread$i)&
    while true
       if [[ -s $TREG ]]
          echo Running:
          cat $TREG
          echo All Done.
       sleep 2

Thursday, May 12, 2016

Last resort remote shell with google drive

When you don't have a public IP for your home network you may use chrome remote desktop, or teamviewer connect remotely. I'm going to demonstrate a simple way which I'm using to control my home server from anywhere.

It is not only simple but also:
  • highly portable where you can issue commands from any device which support google drive such as computer or mobile phone
  • low bandwidth where it is stateless you can decide the execution interval.
  • etc.
The setup is simple your server should have internet access and the device where you issue the commands should have internet access but an end-to-end connection is not required. You will record your command in a document in google drive and the server will read the command execute it and update the results in the same document. 

,--------,         ,---------,           ,--------,
|        |         | google  |           |        |
| server |<------->| drive   |<--------->| device |
|        |         |         |           |        |
‘--------’         ‘---------’           ‘--------’


1. Create a document in google drive called terminal 

2. Download gdrive and copy it to /usr/bin/gdrive and make it executable.

$ sudo chmod +x /usr/bin/gdrive 

3 Authenticate the user where the commands going to be executed.

You will can execute a command line 'gdrive list' so it will provide a url where you can copy paste in the web browser and get a verification code. That code should be pasted so the gdrive will be able access your google drive documents afterward. 

$ gdrive list
Authentication needed
Go to the following url in your browser:

Enter verification code: 4/6EGPmw3...

4. Create a script in your home directory with following code called

cd $HOME
gdrive export --force --mime text/plain $DOCID >& /dev/null
:> $OUT
:> $ERR
RUN=$( tail -1 $FILE | tr -d '\r\357\273\277' )
if [[ $RUN == 'RUN' ]]
   echo >> $FILE
   CMD=$(tail -2 $FILE | head -1 | tr -d '\r\357\273\277')

   #check if the command still running
   CMDSUM=$( echo $CMD | md5sum | awk '{print $1}')
   if [[ ! -f $CMDSUM ]]
      echo $CMD > $CMDSUM
      bash $CMDSUM > $OUT 2> $ERR
      rm $CMDSUM
      cat $OUT >> $FILE

      #if returns nothing, ech OK
      if [[ ! -s $OUT ]]
         echo '-OK-' >> $FILE

      #return errors if exists
      if [[ -s $ERR ]]
         echo '-ERR-' >> $FILE
         cat $ERR >> $FILE

   #write back the details
   gdrive update --name terminal $DOCID $FILE  >& /dev/null

5. Run 'gdirve list' and get the 'document id' and change the script accordingly.

6. Add a cron job where this script will be execute as the given user in each 1m.

$ sudo vim /etc/cron.d/gshell 

# m h dom mon dow user    command
*/1 * * * * <username>  /home/<username>/ >& /dev/null

7. That's it now you can issue your command through google drive

After you complete your command you have to type 'RUN' in order to execute the command you entered above. With this simple script only the last adjacent line of 'RUN' will be executed.
The output will be followed after the 'RUN' statement as given below.
You can keep issuing commands further down and grow the file or you can delete the content and insert command in clean file.
Hope this will be helpful to you...

Wednesday, June 24, 2015

Leap second - 2015 Study

What is it

In order to sync UTC (Coordinated Universal Time) with TAI (International Atomic Time), the day of 30th June 2015 will have 86401s instead of 86400s.

When does it come

Starting from 1972 there have been several leap seconds set. Last one was on 30th June 2012.
Jun 30
Dec 31

Clock Disciplines

Obviously the time syncing information is received through NTP but the system's clock is disciplined by either Kernel of by the NTP itself.

NTP discipline

Normally, the time is slewed if the offset is less than the step threshold, which is 128 ms by default, and stepped if above the threshold. The '-x' option for ntpd that will be discussed below, sets the threshold to 600 s. Since the slew rate of typical kernels is limited to 0.5 ms/s, each second of adjustment requires an amortization interval of 2000 s.   If '-x' option is set the kernel time discipline is disabled.

Kernel discipline

If the NTP program runs on a Linux version which supports kernel discipline (all current kernels supports), then the NTP client-daemon just passes a leap second down to the kernel which handles the leap second. The way that the kernel discipline the clock at leap second insertion is just stepping one second.

How does Linux get to know

There are three methods that Linux well get aware about the leap  second and do the synchronization.

NTP announcement

During the last day before a leap second correction, NTP servers should notify their clients that a leap second will occur. Whether to follow NTP clock discipline or Kernel clock discipline is depend on the NTP client configuration.
The leap second announcement packet content is given below. Note that the 'Leap indicator: +1s'.
Content of the packet using tcpdump
Server, Leap indicator: +1s (64), Stratum 3, poll 8s, precision -6
    Root Delay: 0.188217, Root dispersion: 0.117919, Reference-ID:
      Reference Timestamp:  3644696646.420012712 (2015/06/30 23:44:06)
      Originator Timestamp: 3639291973.586123943 (2015/04/29 10:26:13)
      Receive Timestamp:    3644697550.648012697 (2015/06/30 23:59:10)
      Transmit Timestamp:   3644697550.648012697 (2015/06/30 23:59:10)
        Originator - Receive Timestamp:  +5405577.061888758
        Originator - Transmit Timestamp: +5405577.061888758

ZONE file update

For RHEL, it is also possible to make the reporting time corrected for leap seconds by updating the tzdata package to the latest version available, copying the appropriate file from the /usr/share/zoneinfo/right directory hierarchy to /etc/localtime, and resetting the clock to the correct local time.  The files in /usr/share/zoneinfo/right contain local time information corrected for all leap seconds that have occurred since the beginning of the Epoch on 1970-01-01 00:00:00 UTC.
Example for Colombo
# yum update tzdate
# ln -s /usr/share/zoneinfo/right/Asia/Colombo /etc/localtime

Manually set/unset

It is also possible to trigger leap second using the Kernel clock discipline by using adjtimex() system call. In SLES there is a tool called 'adjtimex' for RHEL we may have to write a program. The Kernel time variable should be set on the day before leap second and it should be disarmed next day so it can be scheduled using cronie.
Crontab entry for manual leap second
59 23 30 6 * root
1 0 1 7 * root

With ntptime (for SLES and RHEL)

# ntptime -s 16
# ntptime -s 0


Set leap second flag
# adjtimex -s16
# adjtimex -s0


example code to set leap second
int main(){
   struct timex tx;
   if(adjtimex(&tx) == -1)                                                        
Compile and run
# ./ls

How to identify if the server is set for leap second

When using NTP discipline

Check if leap announce is recieved
# ntpq -c rl | grep 'leap='
processor="x86_64", system="Linux/3.0.74-0.6.10-default", leap=01,
'leap' Value
00no warning
01last minute of the day has 61 seconds
10last minute of the day has 59 seconds
11unknown (clock unsynchronized)

When using Kernel discipline

With ntptime command in both SLES and RHEL. If the status contains INS flag that means a leap second insertion is set.
# ntptime | grep status
  status 0x50 (INS,UNSYNC),


adjtimex -p print the current values of the kernel time variables.  "status" gives the value of the time_status variable in the kernel. The value is a sum of the following
1PLL updates enabled
2PPS freq discipline enabled
4PPS time discipline enabled
8frequency-lock mode enabled
16inserting leap second
32deleting leap second
64clock unsynchronized
128holding frequency
256PPS signal present
512PPS signal jitter exceeded
1024PPS signal wander exceeded
2048PPS signal calibration error
4096clock hardware fault

# adjtimex -p
         mode: 0
       offset: 0
    frequency: 0
     maxerror: 16000000
     esterror: 16000000
       status: 16
time_constant: 2
    precision: 1
    tolerance: 32768000
         tick: 10000
     raw time:  1431048543s 786257us = 1431048543.786257
 return value = 5


print leap status
int main(){                                                                              
   struct timex tx;                                                               
   if(adjtimex(&tx) == -1){                                                                              
   printf("Kernel leap second flag: ");                                           
   if(tx.status & STA_INS)                                                        
      if(tx.status & STA_DEL)                                                       
         printf("not set\n");                                                         
Command output
Output when not set
$ ./lg
Kernel leap second flag: not set
Output when set to add 1s
$ ./lg
Kernel leap second flag: add

When the leap second is applied with Kernel discipline you will find following message in dmesg
leap insertion message in dmesg
# dmesg | grep -i leap
648.669570] Clock: inserting leap second 23:59:60 UTC

How to Setup a Demo Lab

Setup a NTP server with leap second announcement

  1. Install the ntp server
    SLES: zypper in ntp
    RHEL: yum install ntp
  2. Get the leap file from NIST.
    cd /var/ntp
    sudo wget
  3. Then we need to change settings of ntpd so that it can announce a fake time.
    sudo vi /etc/ntp.conf
  4. Comment all the `server` lines out, and set the server itself as refclock
    fudge stratum 10
  5. Set the leap file
    leapfile "/var/ntp/leap-seconds.list" 
  6. The resulting /etc/ntp.conf file will look like this
    # grep -v ^# /etc/ntp.conf
    fudge stratum 10
    leapfile /var/ntp/leap-seconds.list
    driftfile /var/lib/ntp/drift/ntp.drift
    keys /etc/ntp.keys     
    trustedkey 1
    requestkey 1
  7. Now we set the system time to somewhere a little before a leap second took place, e.g, Jun 30, 2015, and then restart the ntpd daemon.
    export TZ=UTC
    date -s "2015-06-30 23:50"
    service ntp restart
  8. Now we just got back to 10 minutes earlier than a leap-second in the past.

Setup a NTP client with statistics enabled

  1. Install the ntp server
    SLES: zypper in ntp
    RHEL: yum install ntp 
  2. Change the settings of the ntp to use above server.
    sudo vi /etc/ntp.conf
  3. Add ntp server
  4. Create directory for statistics
    mkdir /var/lib/ntp/tmp/
  5. Enable statistics collection - uncomment following lines or add if not available. Note that ntp root is /var/lib/ntp so /tmp/ is equivalent to /var/lib/ntp/tmp/
    statsdir /tmp/        # directory for statistics files
    filegen peerstats  file peerstats  type day enable
    filegen loopstats  file loopstats  type day enable
    filegen clockstats file clockstats type day enable
  6. The resulting ntp.conf file will look like this
    driftfile /var/lib/ntp/drift/ntp.drift     
    logfile /var/log/ntp            
    statsdir /tmp/        # directory for statistics files
    filegen peerstats  file peerstats  type day enable
    filegen loopstats  file loopstats  type day enable
    filegen clockstats file clockstats type day enable
    keys /etc/ntp.keys            
    trustedkey 1                
    requestkey 1
  7. Now sync the time with the ntp server using one-time-mode.
    # ntpdate -q
  8. Now start ntpd daemon
    # service ntp start

NTP configuration with NTP discipline

By default ntpd will run with Kernel discipline mode so when it receive the leap second anouncement it will arm kernel with it. If you want to use NTP discipline mode ntpd should start with '-x' option.
vim /etc/sysconfig/ntp
NTPD_OPTIONS=" -x -g -u ntp:ntp"
vim /etc/sysconfig/ntpd
OPTIONS=" -x -u ntp:ntp -p /var/run/ -g"

Setup a NTP proxy with leap second announcement

NTP server with a drifting clock like the one in the server will provide drifting sync for the client. So setting a ntp server may cause complications to the testing. There is a ntp proxy wrote by Robert Karbowski which is available in github. Download it and make, so you will have two binaries 'ntpproxy' and 'sls'.
  • ntpproxy will add an offset to time of the source ntp server that is somewhere close to the UTC 23:59:59 of 30th June 2015 (default is 600s before leap second) and forward to the client. Clients queries will also transformed accordingly and forward to the server.
  • It will also do the leap announcement to the clients.
  • ntp client configuration will be similar to the one given above.
starting ntp proxy
# ./ntpproxy -s -v
example using internal ntp server:
# ./ntpproxy -s -v


Robert Karbowski also provides a binary called 'sls' we can use this to manually set the kernel time variables for leap second.
  • sls will also set the system time close to the UTC 23:59:59 of 30th June 2015 (default is 600s before leap second)

We can also use adjtimex to set the kernel time variable to demonstrate the leap second.
# export TZ=UTC
# date -s "`date +%Y%m-%d` 23:59:00"
# adjtimex -S16

Check time during test

Following code (clockm) will print the time in 1ms intervals and the printing time format and the functions given below.
:: ::
23:59:50   3100:14:08  526
int main(void)
  char buffer[30],buffer_[30];
  struct timeval  tv1;
  time_t curtime,mtime;
  struct timespec ts1;
     gettimeofday(&tv1, NULL);
     strftime(buffer,30,"%T ",localtime(&curtime));
     printf("%s %ld; ",buffer,tv1.tv_usec/1000);
     strftime(buffer,30,"%T ",localtime(&curtime));
     printf("%s %ld\n",buffer,ts1.tv_nsec/1000000);
  return 0;


Test  the Kernel discipline with clockm

Following script was used to test the kernel discipline when the leap second.
export TZ=UTC
adjtimex -S0
date -s "`date +%Y-%m-%d` 23:59:50"
#date -s "2015-06-30 23:59:50"
adjtimex -S16
./clockm | tee -a leap-$$.log


Clock Discipline
SLES 11 SP33.0.101-0.47.52-defaultKernel...
23:59:59  998; 00:02:10  44
23:59:59  999; 00:02:10  45
00:00:00  0; 00:02:10  46
00:00:00  1; 00:02:10  47
23:59:59  3; 00:02:10  49
23:59:59  4; 00:02:10  50
23:59:59  997; 00:02:11  43
23:59:59  998; 00:02:11  44
23:59:59  999; 00:02:11  45
00:00:00  0; 00:02:11  46
00:00:00  1; 00:02:11  47
00:00:00  2; 00:02:11  48
  • There were two 59th seconds observed
  • 00th second reported between first 59th second and  second 59th second (while switching).
SLES11 SP23.0.13-0.27-defaultKernel...
23:59:59  996; 01:24:29  377
23:59:59  998; 01:24:29  379
23:59:59  999; 01:24:29  380
00:00:00  0; 01:24:29  381
00:00:00  1; 01:24:29  382
00:00:00  2; 01:24:29  383
23:59:59  997; 00:18:02  325
23:59:59  998; 00:18:02  326
23:59:59  999; 00:18:02  327
23:59:59  1; 00:18:02  329
23:59:59  2; 00:18:02  330
23:59:59  3; 00:18:02  331
 23:59:59  996; 00:04:10  177
 23:59:59  998; 00:04:10  178
 23:59:59  999; 00:04:10  179
 23:59:59  0; 00:04:10  180
 23:59:59  1; 00:04:10  181
 23:59:59  2; 00:04:10  182
  • Elapsed two seconds for 59

Test the NTP discipline with clockm


Check if the installed kernel and package versions in RHEL are vulnerable

RedHat has provided a script to perform this called
If the target is vulnerable, you will see output similar to:
$ ./
This system is vulnerable to a performance degradation after the Leap Second Insertion of June 30, 2015.
Please refer to //> for remediation steps.

If the target is not vulnerable, you will see output similar to:
$ ./
Not vulnerable

Test with leap-a-day

RedHat has provided a code leap-a-day.c, for checking a system for 'known hrtimer failure' while leap second was applied.
Clock Discipline
root@vm_perl_dev|SLES11SP2|3.0.13-0.27:~ # ./leap-a-day -s
Setting time to speed up testing
This runs continuously. Press ctrl-c to stop
Setting time to Thu May 14 23:59:50 2015
Scheduling leap second for Fri May 15 00:00:00 2015
Thu May 14 23:59:57 2015 +    195 us (0)    TIME_INS
Thu May 14 23:59:57 2015 + 500303 us (0)    TIME_INS
Thu May 14 23:59:58 2015 +    431 us (0)    TIME_INS
Thu May 14 23:59:58 2015 + 500546 us (0)    TIME_INS
Thu May 14 23:59:59 2015 +    682 us (0)    TIME_INS
Thu May 14 23:59:59 2015 + 500853 us (0)    TIME_INS
Thu May 14 23:59:59 2015 +   1049 us (0)    TIME_OOP
Thu May 14 23:59:59 2015 + 501154 us (0)    TIME_OOP
Fri May 15 00:00:00 2015 +   1268 us (0)    TIME_WAIT
Fri May 15 00:00:00 2015 + 501398 us (0)    TIME_WAIT
Fri May 15 00:00:01 2015 +   1556 us (0)    TIME_WAIT
Fri May 15 00:00:01 2015 + 501687 us (0)    TIME_WAIT
Fri May 15 00:00:02 2015 +   1847 us (0)    TIME_WAIT
ERROR: hrtimer early expiration failure observed.
Leap complete
Setting time to Fri May 15 23:59:50 2015
Scheduling leap second for Sat May 16 00:00:00 2015
Fri May 15 23:59:57 2015 +     51 us (0)    TIME_DEL
Fri May 15 23:59:57 2015 + 500199 us (0)    TIME_DEL
Fri May 15 23:59:58 2015 +    352 us (0)    TIME_DEL
Fri May 15 23:59:58 2015 + 500527 us (0)    TIME_DEL
Sat May 16 00:00:00 2015 +    634 us (0)    TIME_WAIT
Sat May 16 00:00:00 2015 + 500781 us (0)    TIME_WAIT
Sat May 16 00:00:01 2015 +    954 us (0)    TIME_WAIT
Sat May 16 00:00:01 2015 + 501123 us (0)    TIME_WAIT
Sat May 16 00:00:02 2015 +   1332 us (0)    TIME_WAIT
Leap complete
ERROR: hrtimer early expiration failure observed.



SUSERun ntp in slew mode
  1. Stop ntpd
# rcntp stop
2. Reset the kernel status and frequency
# ntptime -s 0 -f 0 
3. Configure ntp with the -x option (ntp slew mode)
Edit /etc/sysconfig/ntp and add -x option as follows
NTPD_OPTIONS=" -x -g -u ntp:ntp"
4. Start ntpd
# rcntp start
Red HatRun ntp in slew mode
  1. Update ntp and ntpdate to the following versions:
  • ntpdate-4.2.6p5-3.el6_6.x86_64
  • ntp-4.2.6p5-3.el6_6.x86_64
2. Stop ntpd
# service ntpd stop
3. Reset the kernel status and frequency
# ntptime -s 0 -f 0 
4. Configure ntp with the -x option (ntp slew mode)
Edit /etc/sysconfig/ntpdOPTIONS=" -x -u ntp:ntp -p /var/run/ -g"
5. Start ntpd
# service ntpd start
SolarisRun ntp in slew mode
  1. Stop NTP service 24 hours prior to the leap second.
# svcadm disable ntp
 2. Add following 2 lines in the /etc/inet/ntp.conf
slewalways yes
disable pll
3. Start ntp
# svcadm enable ntp
Description of the Solution
What Does the Client Have To Do?


Wednesday, April 9, 2014

WiFi Tethering in Linux / Convert your laptop in to an access point

1. install hostapd and dnsmasq to your system

2. Configure hostapd
Sample configuration given below. Change interface, ssid and wpa_passphrase according to your requirement.
$ cat /etc/hostapd/hostapd.conf

3. Create a start/stop script
This script will start, stop, restart  the required services and set the required iptables rules. Please change the IP address and interface name if required. 
$ cat /opt/ 
function start(){
   echo "Starting hostapd"
   hostapd -B /etc/hostapd/hostapd.conf
   sleep 1
   echo "Setting to wlan0"
   ifconfig wlan0 netmask
   echo "Starting dnsmasq..."
   dnsmasq --conf-file --no-hosts --keep-in-foreground --bind-interfaces --except-interface=lo --clear-on-reload --strict-order --listen-address= --dhcp-range=,,60m --dhcp-option=option:router, --dhcp-lease-max=50 --pid-file=/var/run/
   iptables -A FORWARD -d -o wlan0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
   iptables -A FORWARD -s -i wlan0 -j ACCEPT
   iptables -A FORWARD -i wlan0 -o wlan0 -j ACCEPT
   iptables -t nat -A POSTROUTING -s ! -d -j MASQUERADE
   echo 1 > /proc/sys/net/ipv4/ip_forward

function stop(){
   echo "Stopping hostapd..."
   pkill hostapd
   echo "Turn down wlan0..."
   ifconfig wlan0 down
   echo "stoping dnsmasq..."
   pkill -f dnsmasq.**
   iptables -D FORWARD -d -o wlan0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
   iptables -D FORWARD -s -i wlan0 -j ACCEPT
   iptables -D FORWARD -i wlan0 -o wlan0 -j ACCEPT
   iptables -t nat -D POSTROUTING -s ! -d -j MASQUERADE
   echo 0 > /proc/sys/net/ipv4/ip_forward

function status(){
   if [[ $(pgrep -f dnsmasq.**) != '' ]]
      echo "dnsmasq running..."
      echo "dnsmasq not running..."
   if [[ $(pgrep hostapd) != '' ]]
      echo "hostapd running..."
      echo "hostapd not running..."

   echo "ip_forward is set to:"$( cat /proc/sys/net/ipv4/ip_forward)
   echo "Related iptable rules:"
   sudo iptables -nL -t nat | grep 10.42;
   sudo iptables -nL | grep 10.42

case "$1" in
      if [[ $(pgrep -f dnsmasq.**) == '' &&  $(pgrep hostapd) == '' ]] 
      echo "Usage: $0" \
      exit 1

4. That's all. Now you can start, stop and check the status of the access point using this script.
/opt/ start
/opt/ stop
/opt/ restart
/opt/ status

Friday, January 31, 2014

Untethered Jailbreak IPhone 3GS iOS 6.1.3

  1. Backup the iPhone using iTunes
  2. Try this method. 
  3. If that didn't work, then try the following steps.
  4. Upgrade your iPhone 3GS to 6.1.3 using iTunes 
  5. Download Redsn0w 0.9.15b3 from here
  6. Download iOS 6.0.0 IPSW from here
  7. Open up Redsn0w, select Extras then Select IPSW
  8. Select the downloaded iOS 6.0.0 IPSW
  9. Now, from the main screen in Redsn0w just click Jailbreak and make sure you follow the instructions that will appear on the screen. Select install Cydia.
  10. If Cydia is not installed then go to extras hit select IPSW again and click the same 6.0 and then click just boot
  11. After booting connect to the internet through WiFi
  12. From Cydia install p0sixspwn and ultrasn0w.
  13. p0sixspwn will untether the jailbreak. 
  14. ultrasn0w will unlock the provider and allow you to use your SIM.