PXEClusterInstall

From WBITT's Cooker!

(Difference between revisions)

Revision as of 19:45, 14 February 2010

bismilla-hirrahma-nirraheem

Installation using PXE and kickstart My host machine/physical machine is named kworkbee, running Fedora 8 (i386), and VMware server 2.0 on top of it.

My virtual machines are connected on HostOnly network vmnet1. 192.168.0.0/24 . Whereas 192.168.0.1 is the ip of the vmnet1 interface on host machine.

The head node of my cluster is named "headnode" and has an IP of 192.168.0.10 , on it's eth0.

My Virtual machines, part of this beowulf cluster are :-

headnode node1 (MAC address: 00:50:56:00:00:11) node2 (MAC address: 00:50:56:00:00:12)

I am using "redhat" (without quotes), as my root password on all machines.

The /etc/hosts file on my host machine and headnode, looks like:- [root@kworkbee ~]# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 192.168.0.1 kworkbee kworkbee 192.168.0.10 headnode headnode 192.168.0.11 node1 node1 192.168.0.12 node2 node2

On the host server, extracted the ISO file in a directory:

mount -o loop /data/cdimages/CentOS-5.2-i386-bin-DVD.iso /media/loop/

cp -r /media/loop/* /data/cdimages/centos/

vi /etc/httpd/conf.d/centos.conf

Alias /centos /data/cdimages/centos/ <Location /centos>

   Order deny,allow
   Allow from all
   Options +Indexes

</Location>

service httpd restart

Check by opening a browser window, on the host computer. You should get a list of files from the top level directory of the Centos distribution:-

http://localhost/centos/

The same should be accessable from the head node of your cluster, using the address:-

http://192.168.0.1/centos

To ease the pain of installation of vrious software on the headnode, I have edited the yum repositry file on headnode as shown below. Comment out the rest of the file:-

vi /etc/yum.repos.d/CentOS-Base.repo

[base] name=CentOS-$releasever - Base baseurl=http://192.168.0.1/centos/ gpgcheck=0 gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

Install necessary software on the headnode :- DHCP-server TFTP

[root@headnode ~]# yum -y install dhcp tftp-server syslinux

Setup the /etc/dhcpd.conf file as shown here:-

vi /etc/dhcpd.conf

ddns-update-style interim; ignore client-updates;

subnet 192.168.0.0 netmask 255.255.255.0 {

--- default gateway

       option routers                  192.168.0.1;
       option subnet-mask              255.255.255.0;
       option domain-name              "mybeowulf.local";
       option domain-name-servers      192.168.0.10;
       option time-offset              -18000; # Eastern Standard Time
       option ntp-servers              192.168.0.10;
       filename "pxelinux.0";
       range dynamic-bootp 192.168.0.11 192.168.0.20;
       default-lease-time 21600;
       max-lease-time 43200;

       # next-server points to the TFTP/PXE server.

       host node1 {
               filename "pxelinux.0";
               next-server 192.168.0.10;
               hardware ethernet 00:50:56:00:00:11;
               fixed-address 192.168.0.11;
       }

       host node2 {
               filename "pxelinux.0";
               next-server 192.168.0.10;
               hardware ethernet 00:50:56:00:00:12;
               fixed-address 192.168.0.12;
       }

}

Note: The file pxelinux.0 is a special file, which gets installed through the package syslinux. This is the actual pxe boot linux kernel, and is run as soon as the pxe client/ client machine gets a DHCP lease. The actual location of this file is mentioned later in this howto.

[root@headnode ~]# service dhcpd restart Starting dhcpd: [ OK ] [root@headnode ~]# chkconfig --level 35 dhcpd on

Now enable TFTP as well:-

[root@headnode ~]# vi /etc/xinetd.d/tftp service tftp {

       socket_type             = dgram
       protocol                = udp
       wait                    = yes
       user                    = root
       server                  = /usr/sbin/in.tftpd
       server_args             = -s /tftpboot
       disable                 = no
       per_source              = 11
       cps                     = 100 2
       flags                   = IPv4

}

[root@headnode ~]# service xinetd restart Stopping xinetd: [ OK ] Starting xinetd: [ OK ] [root@headnode ~]# chkconfig --level 35 xinetd on [root@headnode ~]#

Next, we need to copy two special files on the host machin, from a special location of distribution media. On the host machine, I checked the list of files to make sure that the files are available:-

[root@kworkbee ~]# ls /data/cdimages/centos/images/pxeboot/ initrd.img README TRANS.TBL vmlinuz [root@kworkbee ~]#

We will copy the vmlinuz and initrd.img files, from this location on the host machine, to /tftpboot directory on the headnode machine :-

On the headnode, we have the CentOS distribution available through HTTP path : http://192.168.0.1:/centos . We will use wget to download these two files from the host machine. I can also use scp, but just to make things interesting, here it is:-

[root@headnode ~]# cd /tftpboot/

[root@headnode tftpboot]# wget http://192.168.0.1/centos/images/pxeboot/vmlinuz

[root@headnode tftpboot]# wget http://192.168.0.1/centos/images/pxeboot/initrd.img

PXE configuration detail:- When a client boots, by default it will look for a configuration file from TFTP, with the same name as it's MAC address. However, adrer trying several options, it will fall back to requesting a default file, with the name "default". This file needs to be in a directory in /tftp of the headnode.

mkdir /tftpboot/pxelinux.cfg

vi /tftpboot/pxelinux.cfg/default

prompt 1 timeout 5 default linux label linux kernel vmlinuz append vga=normal initrd=initrd.img

Now is the time to copy the pxelinux.0 file from the installed location to /tftpboot directory . This file (pxelinux.0) is provided by the syslinux package, which comes with your linux distribution.

cp /usr/lib/syslinux/pxelinux.0 /tftpboot/

Now make sure that all files and direcoties inside /tftpboot is world readable.

chmod +r /tftpboot/* -R

Now you are ready to boot your client. You should see your client getting an IP and a boot file and booting off from the pxe boot image.

While your client boots, you should the following in your /var/log/messages of your headnode:-

tail -f /var/log/messages

Feb 27 19:55:15 beowulf dhcpd: DHCPDISCOVER from 00:50:56:00:00:11 via eth0 Feb 27 19:55:15 beowulf dhcpd: DHCPOFFER on 192.168.0.11 to 00:50:56:00:00:11 via eth0 Feb 27 19:55:17 beowulf dhcpd: Dynamic and static leases present for 192.168.0.11. Feb 27 19:55:17 beowulf dhcpd: Remove host declaration node1 or remove 192.168.0.11 Feb 27 19:55:17 beowulf dhcpd: from the dynamic address pool for 192.168.0/24 Feb 27 19:55:17 beowulf dhcpd: DHCPREQUEST for 192.168.0.11 (192.168.0.10) from 00:50:56:00:00:11 via eth0 Feb 27 19:55:17 beowulf dhcpd: DHCPACK on 192.168.0.11 to 00:50:56:00:00:11 via eth0 Feb 27 19:55:17 beowulf in.tftpd[12476]: tftp: client does not accept options

By doing this, you have managed to start up the interactive installation . Congratulation!

For automated KickStart based setups, you need to do the following additional steps.

First you need a kickstart file. You can use the minimal kickstart file from your headnode! How ? Well, when you installed your headnode, the installer created a file anaconda-ks.cfg in your /root directory. You can use this file, modify it a bit and use it as the kickstart file for your compute nodes.

[root@headnode ~]# cp anaconda-ks.cfg compute-ks.cfg

[root@headnode ~]# vi compute-ks.cfg

Kickstart file automatically generated by anaconda.

install

My centos distribution is on the hostmachine (kworkbee)(192.168.0.1),
, not on the headnode.

url --url http://192.168.0.1/centos lang en_US.UTF-8 keyboard us network --device eth0 --bootproto dhcp --hostname node rootpw --iscrypted $1$t7dSrF04$Ea4kcb4QFbC3JdZmVyTTA/ firewall --disabled authconfig --enableshadow --enablemd5 selinux --disabled timezone Asia/Riyadh zerombr yes bootloader --location=mbr --driveorder=sda

The following is the partition information you requested
Note that any partitions you deleted are not expressed
here so unless you clear all partitions first, this is
not guaranteed to work

clearpart --all --initlabel part / --fstype ext3 --size=1 --grow part swap --size=256

reboot %packages @base

Copy this file to the document root of your web server, and make it world readable:-

cp /root/compute-ks.cfg /var/www/html/ chmod +r /var/www/html/compute-ks.cfg

Edit the tftpboot file again and add extra options:-

[root@headnode ~]# vi /tftpboot/pxelinux.cfg/default

prompt 1 timeout 5 default linux label linux

       kernel vmlinuz
       append vga=normal initrd=initrd.img ip=dhcp ksdevice=eth0 ks=http://192.168.0.10/compute-ks.cfg

You need to make sure that httpd is running on your head node otherwise the installer will not be able to access this file. Another option is to copy this compute-ks.cfg file to the document root of your host machine (192.168.0.1), in /var/www/html directory.

[root@headnode ~]# service httpd restart

Time to test this setup. On the head node, open up /var/log/messages and /var/log/httpd/access_log files in separate terminals. You should get an entry in your apache access log file, when the installer gets the kickstart file, from the headnode.

For the logs of package retrieval during the actual installation, you should check apache access log on the hostmachine.

[root@headnode ~]# tail -f /var/log/httpd/access_log 192.168.0.11 - - [27/Feb/2009:21:06:50 +0300] "GET /compute-ks.cfg HTTP/1.0" 200 680 "-" "anacona/11.1.2.113"

[root@headnode ~]# tail -f /var/log/messages Feb 27 21:06:51 beowulf dhcpd: DHCPDISCOVER from 00:50:56:00:00:11 via eth0 Feb 27 21:06:51 beowulf dhcpd: DHCPOFFER on 192.168.0.11 to 00:50:56:00:00:11 via eth0 Feb 27 21:06:51 beowulf dhcpd: Dynamic and static leases present for 192.168.0.11. Feb 27 21:06:51 beowulf dhcpd: Remove host declaration node1 or remove 192.168.0.11 Feb 27 21:06:51 beowulf dhcpd: from the dynamic address pool for 192.168.0/24 Feb 27 21:06:51 beowulf dhcpd: DHCPREQUEST for 192.168.0.11 (192.168.0.10) from 00:50:56:00:00:11 via eth0 Feb 27 21:06:51 beowulf dhcpd: DHCPACK on 192.168.0.11 to 00:50:56:00:00:11 via eth0

Try logging in to a node after installation. The screenshot below shows that regardless of the node we log into, we are shown "node" as the hostname of the node. This is because we fixed the hostname as node in the kickstart file. This is a limitation in this type of installation. In case you are installing more than one nodes of a cluster, using such method, you either create separate pxe files and related separate kickstart files, through some sort of script and install nodes using that method. Or, you can manually change the node name after they are installed.

http://www.debian-administration.org/articles/478 http://linux-sys.org/internet_serving/pxeboot.html

This can be observed manually by doing the following manual steps.

1) rename the file /tftpboot/pxelinux.cfg/default to 01-<MAC address of any one of your node> i.e. (e.g. node1)

[root@headnode pxelinux.cfg]# mv /tftpboot/pxelinux.cfg/default /tftpboot/pxelinux.cfg/01-00-50-56-00-00-11

The "01", before the MAC address represents a hardware type of ethernet.

Now as you note that you don't have a default file in your tftp setup any more. Only node1 should be able to boot and install from a pxe image properly. Node2, should fail. This can be seen from the screenshot (pxe-boot-7.png) below:-

So you need to develop a mechanism to automate this all, for your cluster. And for the sake of automating a simple task, such as initial installation, there are management tools / software, such as ROCKS, OSCAR, Scali / Platform, etc.

With the help of a little scripting , and the availability of hostnames, mac addresses and IP address range, we can create multiple pxe boot files and related multiple KickStart files. Each pxe file will have an entry against it's related ks file only. A simple example is show below:-

First, the PXE file for node1:-

[root@headnode ~]# vi /tftpboot/pxelinux.cfg/01-00-50-56-00-00-11

prompt 1 timeout 5 default linux label linux

       kernel vmlinuz
       append vga=normal initrd=initrd.img ip=dhcp ks=http://192.168.0.10/node1-ks.cfg

And then, the KickStart file for node1. Notice the different network line for both static IP and hostname :-

[root@headnode ~]# vi /var/www/html/node1-ks.cfg

Kickstart file automatically generated by anaconda.

install

My centos distribution is on the hostmachine (kworkbee)(192.168.0.1),
, not on the headnode.

url --url http://192.168.0.1/centos lang en_US.UTF-8 keyboard us network --device eth0 --bootproto static --ip 192.168.0.11 --netmask 255.255.255.0 --gateway 192.168.0.10 --nameserver 192.168.0.10 --hostname node1 rootpw --iscrypted $1$t7dSrF04$Ea4kcb4QFbC3JdZmVyTTA/ firewall --disabled authconfig --enableshadow --enablemd5 selinux --disabled timezone Asia/Riyadh zerombr yes bootloader --location=mbr --driveorder=sda

The following is the partition information you requested
Note that any partitions you deleted are not expressed
here so unless you clear all partitions first, this is
not guaranteed to work

clearpart --all --initlabel part / --fstype ext3 --size=1 --grow part swap --size=256

reboot %packages @base

[root@headnode .ssh]# ssh-keygen -t dsa -f /root/.ssh/id_dsa -C -N Generating public/private dsa key pair. Your identification has been saved in /root/.ssh/id_dsa. Your public key has been saved in /root/.ssh/id_dsa.pub. The key fingerprint is: 8f:90:a2:02:82:e3:7a:4b:55:a7:79:cc:14:87:3b:d9 [root@headnode .ssh]#

[root@headnode .ssh]# cat id_dsa.pub >> authorized_keys [root@headnode .ssh]# cat id_rsa.pub >> authorized_keys

Now try logging in to this machine:-

[root@headnode .ssh]# ssh localhost Last login: Sat Mar 7 13:12:09 2009 from localhost.localdomain [root@headnode ~]#

As you can see, it works without asking for password. Good. Now lets copy the private files and the public files to the .ssh directory of the nodes. We can put them in a special directory named ssh in our webroot and can get them through wget on the node, during %post of the kickstart . This way all nodes will have a single ssh private and public file. Not much hassle.

[root@headnode .ssh]# mkdir /var/www/html/ssh

[root@headnode .ssh]# cp /root/.ssh/id_* /var/www/html/ssh/

[root@headnode .ssh]# cp /root/.ssh/authorized_keys /var/www/html/ssh/

[root@headnode .ssh]# chmod +r /var/www/html/ssh/*

[root@headnode .ssh]# ls -l /var/www/html/ssh/ total 20 -rw-r--r-- 1 root root 972 Mar 7 13:18 authorized_keys -rw-r--r-- 1 root root 672 Mar 7 13:17 id_dsa -rw-r--r-- 1 root root 590 Mar 7 13:17 id_dsa.pub -rw-r--r-- 1 root root 1675 Mar 7 13:17 id_rsa -rw-r--r-- 1 root root 382 Mar 7 13:17 id_rsa.pub [root@headnode .ssh]#

Now lets write down steps of setting up a correct .ssh directory on the node.

mkdir /root/.ssh chmod 0700 /root/.ssh cd /root/.ssh wget http://192.168.0.10/ssh/id_dsa wget http://192.168.0.10/ssh/id_dsa.pub wget http://192.168.0.10/ssh/id_rsa wget http://192.168.0.10/ssh/id_rsa.pub wget http://192.168.0.10/ssh/authorized_keys

chmod 0600 /root/.ssh/*

cd

Lets test:

[root@headnode .ssh]# ssh root@node1 Last login: Sat Mar 7 13:27:13 2009 from headnode [root@node1 ~]#

Great ! Let's check the other way round.

[root@node1 .ssh]# ssh root@headnode Last login: Sat Mar 7 13:13:11 2009 from localhost.localdomain [root@headnode ~]#

This works great too!

Note: This excercise assumes that RSA HOST Keys were scanned for each host and saved in /etc/ssh/ssh_known_hosts . However, it doesn't make sense. You see, in the beginning there will be only one host RSA key in the /etc/ssh/ssh_known_hosts on the headnode. And that would be the entry for headnode only. And we also see that we could not know the RSA host key of any nodes which are not installed yet. So, one way to do it is to make this ssh_known_hosts file available to the node as well over http. We will copy this file from the headnode to the client in the %post of client kickstart.

But, how would the headnode know that it is time that a particular node is done installation and it can generate it's ssh-host-key and add it to its /etc/ssh/ssh_known_hosts ? I wonder how my company is doing it ?

Remember we are doing this manually anyway. I mean the node restart one by one . We are not doing it through any management software as yet. So we need to manually add the ssh-host-key of each node to the /etc/ssh/ssh_known_hosts file on the server, at the end of installation of each node.

We can still automate it somhow. That is, we can put the ssh host key of this computer on the headnode as soon as this node is isntalled. Through a cron or something, we will add the keys of all nodes, which are installed , to the ssh_known_hosts file on headnode.

Or we can constantly monitor some log file to check when a node sends completion signal and we can initiate its key generation process.

The following four commands setup the headnode's ssh-host-key on the node.

[root@headnode .ssh]# ssh-keyscan -t rsa headnode > /var/www/html/ssh/ssh_known_hosts chmod +r /var/www/html/ssh/ssh_known_hosts

On the node side:-

[root@node1 ssh]# rm ssh_known_hosts* -f [root@node1 ssh]# wget http://192.168.0.10/ssh/ssh_known_hosts

Then, I would use the following to add the ssh-host-key of the newly installed compute node to the ssh_known_hosts file on the headnode:-

[root@node1 ssh]# SSH_SCAN_KEY=$(ssh-keyscan -t rsa `hostname -s`)

node1 SSH-2.0-OpenSSH_4.3

[root@node1 ssh]#

[root@node1 ssh]# ssh headnode "echo $SSH_SCAN_KEY >> /tmp/hosts.txt" [root@node1 ssh]#

Let's send the FQDN key as well:-

[root@node1 ssh]# SSH_SCAN_KEY=$(ssh-keyscan -t rsa `hostname`)

node1.mybeowulf.local SSH-2.0-OpenSSH_4.3

[root@node1 ssh]#

[root@node1 ssh]# ssh headnode "echo $SSH_SCAN_KEY >> /tmp/hosts.txt" [root@node1 ssh]#

Let's check on the server side:-

[root@headnode ssh]# cat /tmp/hosts.txt node1 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzKIg/MmPJzPoQxBWRN8G8ZGad74EqXRyR1T6EXWXQ+xvSKZmI6CuvExBuXoKCBVJ/TzTQ5x46c4fM2+3aU0xTpupzCGhrpcI+21ITwhJjlaF6Kc0CGhyTG8ztftxIdcBus0rW8VkSvVbLnMTDPQstHAVvrSqahoBfLCAWqLnWcJ8+BqenFtPI9Tvq6Dj+Ilx+ukNiGoS7+ng43WGWMHWP4LtGeI/628Hzt23WCjSLL+HqzoUF3u8ouwZlPiYP8BbUXOoTG9XME9M4Oiny0X6LoHMf0lNO89dlFpllRL3ZzURXPO+bT4KiR/Juo645JhTDi0Y7Nk6MToML0ji00yKVw== node1.mybeowulf.local ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzKIg/MmPJzPoQxBWRN8G8ZGad74EqXRyR1T6EXWXQ+xvSKZmI6CuvExBuXoKCBVJ/TzTQ5x46c4fM2+3aU0xTpupzCGhrpcI+21ITwhJjlaF6Kc0CGhyTG8ztftxIdcBus0rW8VkSvVbLnMTDPQstHAVvrSqahoBfLCAWqLnWcJ8+BqenFtPI9Tvq6Dj+Ilx+ukNiGoS7+ng43WGWMHWP4LtGeI/628Hzt23WCjSLL+HqzoUF3u8ouwZlPiYP8BbUXOoTG9XME9M4Oiny0X6LoHMf0lNO89dlFpllRL3ZzURXPO+bT4KiR/Juo645JhTDi0Y7Nk6MToML0ji00yKVw== [root@headnode ssh]#

Alhumdulillah. As you can see, the test is successful. So I will use /etc/ssh/ssh_known_hosts file instead of using /tmp/hosts.txt .

[root@node1 ssh]# SSH_SCAN_KEY=$(ssh-keyscan -t rsa `hostname -s`)

node1 SSH-2.0-OpenSSH_4.3

[root@node1 ssh]# ssh headnode "echo $SSH_SCAN_KEY >> /etc/ssh/ssh_known_hosts" [root@node1 ssh]# SSH_SCAN_KEY=$(ssh-keyscan -t rsa `hostname`)

node1.mybeowulf.local SSH-2.0-OpenSSH_4.3

[root@node1 ssh]# ssh headnode "echo $SSH_SCAN_KEY >> /etc/ssh/ssh_known_hosts" [root@node1 ssh]#

On the server:- [root@headnode ssh]# cat /etc/ssh/ssh_known_hosts node1 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzKIg/MmPJzPoQxBWRN8G8ZGad74EqXRyR1T6EXWXQ+xvSKZmI6CuvExBuXoKCBVJ/TzTQ5x46c4fM2+3aU0xTpupzCGhrpcI+21ITwhJjlaF6Kc0CGhyTG8ztftxIdcBus0rW8VkSvVbLnMTDPQstHAVvrSqahoBfLCAWqLnWcJ8+BqenFtPI9Tvq6Dj+Ilx+ukNiGoS7+ng43WGWMHWP4LtGeI/628Hzt23WCjSLL+HqzoUF3u8ouwZlPiYP8BbUXOoTG9XME9M4Oiny0X6LoHMf0lNO89dlFpllRL3ZzURXPO+bT4KiR/Juo645JhTDi0Y7Nk6MToML0ji00yKVw== node1.mybeowulf.local ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzKIg/MmPJzPoQxBWRN8G8ZGad74EqXRyR1T6EXWXQ+xvSKZmI6CuvExBuXoKCBVJ/TzTQ5x46c4fM2+3aU0xTpupzCGhrpcI+21ITwhJjlaF6Kc0CGhyTG8ztftxIdcBus0rW8VkSvVbLnMTDPQstHAVvrSqahoBfLCAWqLnWcJ8+BqenFtPI9Tvq6Dj+Ilx+ukNiGoS7+ng43WGWMHWP4LtGeI/628Hzt23WCjSLL+HqzoUF3u8ouwZlPiYP8BbUXOoTG9XME9M4Oiny0X6LoHMf0lNO89dlFpllRL3ZzURXPO+bT4KiR/Juo645JhTDi0Y7Nk6MToML0ji00yKVw== [root@headnode ssh]#

Now, I can ssh into my nodes, without password:-

[root@headnode ssh]# ssh node1 Last login: Sat Mar 7 13:27:38 2009 from headnode [root@node1 ~]#

Alhumdulillah.

Or should I setup Rlogin first ? For Rlogin, we need to have rsh-server package installed on compute nodes. And to have it two-way, we need to have it on both compute nodes and the headnode.

Put this in post:-

[root@headnode ssh]# yum -y install rsh-server

[root@headnode ssh]# perl -pi -e 's/= yes/= no/' /etc/xinetd.d/rlogin

[root@headnode ssh]# cat /etc/xinetd.d/rlogin

default: on
description: rlogind is the server for the rlogin(1) program. The server \
provides a remote login facility with authentication based on \
privileged port numbers from trusted hosts.

service login {

       socket_type             = stream
       wait                    = no
       user                    = root
       log_on_success          += USERID
       log_on_failure          += USERID
       server                  = /usr/sbin/in.rlogind
       disable                 = no

}

[root@headnode ssh]# perl -pi -e 's/= yes/= no/' /etc/xinetd.d/rexec

[root@headnode ssh]# cat /etc/xinetd.d/rexec

default: off
description: Rexecd is the server for the rexec(3) routine. The server \
provides remote execution facilities with authentication based \
on user names and passwords.

service exec {

       socket_type             = stream
       wait                    = no
       user                    = root
       log_on_success          += USERID
       log_on_failure          += USERID
       server                  = /usr/sbin/in.rexecd
       disable                 = no

}

[root@headnode ssh]# perl -pi -e 's/= yes/= no/' /etc/xinetd.d/rsh [root@headnode ssh]# cat /etc/xinetd.d/rsh

default: on
description: The rshd server is the server for the rcmd(3) routine and, \
consequently, for the rsh(1) program. The server provides \
remote execution facilities with authentication based on \
privileged port numbers from trusted hosts.

service shell {

       socket_type             = stream
       wait                    = no
       user                    = root
       log_on_success          += USERID
       log_on_failure          += USERID
       server                  = /usr/sbin/in.rshd
       disable                 = no

} [root@headnode ssh]#

service xinetd restart

Same on the node:-

[root@node1 ssh]# perl -pi -e 's/= yes/= no/' /etc/xinetd.d/rlogin [root@node1 ssh]# perl -pi -e 's/= yes/= no/' /etc/xinetd.d/rexec [root@node1 ssh]# perl -pi -e 's/= yes/= no/' /etc/xinetd.d/rsh [root@node1 ssh]# service xinetd restart Stopping xinetd: [FAILED] Starting xinetd: [ OK ] [root@node1 ssh]# chkconfig --level 35 xinetd on [root@node1 ssh]#

setup hosts.equiv files for r* commands.

Server:-

What about YUM repository on the nodes? Put the following in the %post:-

[root@node1 ssh]# cat > /etc/yum.repos.d/CentOS-Base.repo << EOF [base] name=CentOS-$releasever - Base baseurl=http://192.168.0.1/centos/ gpgcheck=1 gpgkey=http://192.168.0.1/centos/RPM-GPG-KEY-CentOS-5 EOF

OK. Now, lets setup MPI on this cluster.

We need a central storage for storing MPI programs and user home directories.

on the server:-

mkdir /cluster mkdir /cluster/mpiuser

vi /etc/exports /cluster *(rw,no_root_squash,sync)

service nfs restart chkconfig --level 35 nfs on

We need to mount this directory on all cluster nodes.

[root@node1 ~]# mkdir /cluster [root@node1 ~]# mount -t nfs headnode:/cluster /cluster

[root@node2 ~]# mkdir /cluster [root@node2 ~]# mount -t nfs headnode:/cluster /cluster

Put the mount request in /etc/fstab of all the compute nodes. Also put the same in %post of compute.ks .

Now we need an MPI user with same userID on all nodes. This can be done manually as following, or through NIS.

[root@headnode ~]# groupadd -g 600 mpiuser [root@headnode ~]# useradd -u 600 -g 600 -c "MPI user" -d /cluster/mpiuser mpiuser

[root@headnode ~]# ls -l /cluster/ total 4 drwx------ 3 mpiuser mpiuser 4096 Mar 8 09:16 mpiuser [root@headnode ~]#

And on all compute nodes as well :-

groupadd -g 600 mpiuser useradd -u 600 -g 600 -c "MPI user" -d /cluster/mpiuser mpiuser

Next we need ssh equivalence for mpiuser on all nodes. We already know that they have common home mounted on each node, as /cluster/mpiuser. So we just need to generate ssh keys for them and put the public key in the authorized_keys file only on headnode. By doing that, we would setup the ssh equivalence automateically.

[root@headnode ~]# su - mpiuser

[mpiuser@headnode ~]$ ssh-keygen -t rsa -C -N -f /cluster/mpiuser/.ssh/id_rsa Generating public/private rsa key pair. Created directory '/cluster/mpiuser/.ssh'. Your identification has been saved in /cluster/mpiuser/.ssh/id_rsa. Your public key has been saved in /cluster/mpiuser/.ssh/id_rsa.pub. The key fingerprint is: 1e:93:94:b8:f3:51:d7:84:31:2a:66:28:ad:23:ab:e8

[mpiuser@headnode ~]$ ssh-keygen -t dsa -C -N -f /cluster/mpiuser/.ssh/id_dsa Generating public/private dsa key pair. Your identification has been saved in /cluster/mpiuser/.ssh/id_dsa. Your public key has been saved in /cluster/mpiuser/.ssh/id_dsa.pub. The key fingerprint is: 63:ef:8b:62:94:ea:88:83:c9:73:78:5b:f7:a0:0f:08

Now lets copy the public key to the authorized_keys.

[mpiuser@headnode ~]$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys

[mpiuser@headnode ~]$ cat .ssh/id_dsa.pub >> .ssh/authorized_keys

[mpiuser@headnode ~]$ chmod 600 .ssh/*

Try logging on to the node1 as mpiuser:-

[mpiuser@headnode ~]$ ssh mpiuser@node1 Warning: Permanently added the RSA host key for IP address '192.168.0.11' to the list of known hosts. [mpiuser@node1 ~]$

Great! Alhumdulillah!

Ok. Now we need GCC on all nodes (headnode+compute).

yum -y install gcc

After that, we need to download MPI version 2, also known as MPICH. The site is:-

http://www.mcs.anl.gov/research/projects/mpich2

Download on headnode and compile it in the shared location /cluster/mpich2 .

cd /cluster

wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.0.8/mpich2-1.0.8.tar.gz

tar xzf mpich2-1.0.8.tar.gz

cd mpich2-1.0.8

mkdir /cluster/mpich2

./configure --prefix=/cluster/mpich2

On a VMware machine, the configuration part takes 2-3 minutes.

make

On a VMware machine, the compilation part takes 2-3 minutes.

make install

Alright, now we need to define certain environment variables in the .bashrc or .bash_profile of the mpiuser.\

vi /cluster/mpiuser/.bash_profile ... PATH=$PATH:$HOME/bin:/cluster/mpich2/bin LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/cluster/mpich2/lib

I personally think that the following two lines are totally useless:-

   Next we run this command in order to define MPICH installation path to SSH.
   mpiu@ub0:~$ sudo echo /mirror/mpich2/bin >> /etc/environment

Lets login as mpiuser and see if our mpi executables are found when needed:-

su - mpiuser

[mpiuser@headnode ~]$ which mpd /cluster/mpich2/bin/mpd

[mpiuser@headnode ~]$ which mpiexec /cluster/mpich2/bin/mpiexec

[mpiuser@headnode ~]$ which mpirun /cluster/mpich2/bin/mpirun

Setup MPD:- MPD is MPI Daemon. We need to create a file named mpd.hosts in mpiuser's home directory, and put in the names of our compute nodes. ( I am using headnode as compute node as well).

vi /cluster/mpiuser/mpd.hosts headnode node1 node2

We also need to have a secrets file for the cluster:-

vi /cluster/mpiuser/.mpd.conf secretword=redhat

Tighten the permissions:-

chmod 0600 /cluster/mpiuser/.mpd.conf

Now run the following sequence of commands to check if things are working:-

On the headnode:-

mpd & sleep 2 mpdtrace mpdallexit

This should give you the following output. Notice the hostname returned by the mpdtrace command:-

[mpiuser@headnode ~]$ mpd& [1] 19235 [mpiuser@headnode ~]$ mpdtrace headnode [mpiuser@headnode ~]$ mpdallexit [mpiuser@headnode ~]$

Here is an interesting check. I intentionally shutdown node2 and then checked, what MPD reports:-

[mpiuser@headnode ~]$ mpdboot -n 3 --chkuponly checking node1 checking node2 these hosts are down; exiting ['node2'] [mpiuser@headnode ~]$

Lets try booting MPD on all three nodes:-

[mpiuser@headnode ~]$ mpdboot -n 3 mpdboot_headnode (handle_mpd_output 406): from mpd on node2, invalid port info: no_port

Failed! Ok. Lets boot MPD on two nodes only:-

[mpiuser@headnode ~]$ mpdboot -n 2

mpdtrace should return the name of hosts mpd is successfully running on:-

[mpiuser@headnode ~]$ mpdtrace headnode node1 [mpiuser@headnode ~]$

[mpiuser@headnode ~]$ mpdtrace -l headnode_52307 (192.168.0.10) node1_38651 (192.168.0.11)

So far so good. Lets execute a sample program provided to us in the examples directory in mpch2 source code directory:-

[mpiuser@headnode cluster]$ cd /cluster/mpich2-1.0.8/examples/

There is a compiled program in this directory. The othere programs need compiling with mpimake, NOT with simple make.

[mpiuser@headnode examples]$ ls -l total 968 -rw-r--r-- 1 3714 311 678 Nov 3 2007 child.c -rwxr-xr-x 1 root root 577390 Mar 8 10:17 cpi -rw-r--r-- 1 3714 311 1515 Nov 3 2007 cpi.c -rw-r--r-- 1 root root 1964 Mar 8 10:17 cpi.o ... ...

Lets run one process of mpiexec. mpiexec will automatically select any one node to do that.:-

[mpiuser@headnode examples]$ mpiexec -n 1 ./cpi Process 0 of 1 is on headnode pi is approximately 3.1415926544231341, Error is 0.0000000008333410 wall clock time = 0.000014 [mpiuser@headnode examples]$

Lets run two processes of mpiexec. mpiexec will automatically select any two nodes to do that.:-

[mpiuser@headnode examples]$ mpiexec -n 2 ./cpi Process 0 of 2 is on headnode Process 1 of 2 is on node1.mybeowulf.local pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.001619 [mpiuser@headnode examples]$

Lets run two processes of mpiexec. mpiexec will automatically place two processes (out of total of four), on each node, as we have only two nodes:-

[mpiuser@headnode examples]$ mpiexec -n 4 ./cpi Process 0 of 4 is on headnode Process 1 of 4 is on node1.mybeowulf.local Process 2 of 4 is on headnode Process 3 of 4 is on node1.mybeowulf.local pi is approximately 3.1415926544231239, Error is 0.0000000008333307 wall clock time = 0.005809

The wall time has increased, by the increase in number of processes. It should have decreased, you must be thinking. You are right! But the hardware is not! Notice that these machines, are on a single laptop computer, created inside vmware. As soon as we increase nodes. The same single CPU is divided and shared between the compute nodes. Effectively decreasing the compute power of each node, all of a sudden. On real hardware based compute nodes. This time, WILL decrease, as each process will have a full CPU to itself and thus will take lesser time.

You can run other examples as well, by compiling them:-

There is a file named, icpi. Lets compile that and run that. [icpi is Interactive version of cpi].

[mpiuser@headnode examples]$ ls -l total 968 ... ... -rw-r--r-- 1 root root 1964 Mar 8 10:17 cpi.o -rw-r--r-- 1 3714 311 4469 Nov 3 2007 cpi.vcproj drwxr-xr-x 2 3714 311 4096 Mar 8 10:12 cxx drwxr-xr-x 2 3714 311 4096 Oct 24 20:31 developers drwxr-xr-x 2 3714 311 4096 Mar 8 10:12 f90 -rw-r--r-- 1 3714 311 1892 Nov 3 2007 icpi.c ... ...

Compile:-

[mpiuser@headnode examples]$ mpicc -o /cluster/mpiuser/icpi /cluster/mpich2-1.0.8/examples/icpi.c

Execute:-

cd ~

[mpiuser@headnode ~]$ mpiexec -n 2 ./icpi Enter the number of intervals: (0 quits) 1000 pi is approximately 3.1415927369231258, Error is 0.0000000833333327 wall clock time = 0.008686 Enter the number of intervals: (0 quits) 100000000 pi is approximately 3.1415926535900001, Error is 0.0000000000002069 wall clock time = 1.812169 Enter the number of intervals: (0 quits) 0 [mpiuser@headnode ~]$

Terminate the MPD daemon:-

mpdallexit

By using these simple examples, we have seen how we can setup and run MPI and MPI based programs. Alhumdulillah.

Linpack

Linpack needs mpi/lam or mpich or openmpi installed on the system. User equivalence should also be setup. This is what we have already setup in the steps above.

We need g77, gcc and related compilers, on all nodes:-

yum -y install gcc compat-gcc-34-g77

Next we need to download GOTO BLAS library. This is available at www.tacc.utexas.edu/resources/software/software.php . It needs trivial user registration, which you should go through.

After downloading it, extract it and compile it.

[as root] chown mpiuser:mpiuser /cluster -R

su - mpiuser tar xzf GotoBLAS-1.26.tar.gz cd ~/GotoBLAS

Some guides may ask you to uncomment a line in Makefile.rule

You may want to uncomment the following line, (around line # 14 or 16), so it shows:

F_COMPILER = G77

Please node that in GotoBLAS-1.26, the comments in the Makefile.rule say that if the line is not uncommented, it will use g77 anyway. So no need to change anything here.

The README file tells us to run :- ./quickbuild.32bit

... ... ./gensymbol linktest _ 1 > linktest.c gcc -O2 -D_GNU_SOURCE -Wall -fPIC -DF_INTERFACE_GFORT -DMAX_CPU_NUMBER=1 -DNUM_BUFFERS=$2*1$ -DEXPRECISION -m128bit-long-double -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DBUNDERSCORE=_ -DNEEDBUNDERSCORE -I.. -DARCH_X86 -DCORE2 -DL1_CODE_SIZE=32768 -DL1_CODE_ASSOCIATIVE=8 -DL1_CODE_LINESIZE=64 -DL1_DATA_SIZE=32768 -DL1_DATA_ASSOCIATIVE=8 -DL1_DATA_LINESIZE=64 -DL2_SIZE=4194304 -DL2_ASSOCIATIVE=8 -DL2_LINESIZE=64 -DITB_SIZE=4096 -DITB_ASSOCIATIVE=4 -DITB_ENTRIES=128 -DDTB_SIZE=4096 -DDTB_ASSOCIATIVE=4 -DDTB_ENTRIES=256 -DHAVE_CMOV -DHAVE_MMX -DHAVE_SSE -DHAVE_SSE2 -DHAVE_SSE3 -DHAVE_SSSE3 -DHAVE_CFLUSH -DHAVE_HIT=1 -DNUM_SHAREDCACHE=1 -DNUM_CORES=1 -DCORE_CORE2 -w -o linktest linktest.c ../libgoto_core2-r1.26.so -lm -lm && echo OK. OK. rm -f linktest

Done. This library is compiled with following conditions.

Binary ... 32bit Fortran ... GFORTRAN

Then run make:-

[mpiuser@headnode GotoBLAS]$ make ... ... DCNAME=zpotri -DBUNDERSCORE=_ -DNEEDBUNDERSCORE -I../.. -DARCH_X86 -DCORE2 -DL1_CODE_SIZE=32768 -DL1_CODE_ASSOCIATIVE=8 -DL1_CODE_LINESIZE=64 -DL1_DATA_SIZE=32768 -DL1_DATA_ASSOCIATIVE=8 -DL1_DATA_LINESIZE=64 -DL2_SIZE=4194304 -DL2_ASSOCIATIVE=8 -DL2_LINESIZE=64 -DITB_SIZE=4096 -DITB_ASSOCIATIVE=4 -DITB_ENTRIES=128 -DDTB_SIZE=4096 -DDTB_ASSOCIATIVE=4 -DDTB_ENTRIES=256 -DHAVE_CMOV -DHAVE_MMX -DHAVE_SSE -DHAVE_SSE2 -DHAVE_SSE3 -DHAVE_SSSE3 -DHAVE_CFLUSH -DHAVE_HIT=1 -DNUM_SHAREDCACHE=1 -DNUM_CORES=1 -DCORE_CORE2 -DCOMPLEX -DDOUBLE zpotri.c -o zpotri.o ar -ru ../../libgoto_core2-r1.26.a spotri.o dpotri.o cpotri.o zpotri.o make[2]: Leaving directory `/cluster/GotoBLAS/lapack/potri' make[1]: Leaving directory `/cluster/GotoBLAS/lapack'

Next, download LinPack (also known as HPL or xHPL), from www.netlib.org/benchmark/hpl . There are now two versions available on this site:-

hpl-2.0.tar.gz (updated September 10, 2008) and hpl.tgz (updated January 20, 2004)

I downloaded both of them. First I will check hpl.tgz .

cd /cluster tar xzf hpl.tgz cd /cluster/hpl

[mpiuser@headnode hpl]$ cp setup/Make.Linux_PII_FBLAS_gm .

Now, I need some information first.

What is my GCC version? 4.1.2

[mpiuser@node1 ~]$ gcc -v Using built-in specs. Target: i386-redhat-linux Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=i386-redhat-linux Thread model: posix gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)

Do I have my GCC related files on the OS? yes:-

[mpiuser@node1 ~]$ ls /usr/lib/gcc/i386-redhat-linux/4.1.2/ crtbegin.o crtend.o include libgcc_s.so libgomp.so crtbeginS.o crtendS.o libgcc.a libgcov.a libgomp.spec crtbeginT.o crtfastmath.o libgcc_eh.a libgomp.a SYSCALLS.c.X [mpiuser@node1 ~]$

Time to edit this file:-

[mpiuser@headnode hpl]$ vi Make.Linux_PII_FBLAS_gm ... ... TOPdir = $(HOME)/hpl INCdir = $(TOPdir)/include BINdir = $(TOPdir)/bin/$(ARCH) LIBdir = $(TOPdir)/lib/$(ARCH)

HPLlib = $(LIBdir)/libhpl.a ... ... MPdir = MPinc = MPlib = ... ...

LAdir = $(HOME)/GotoBLAS LAinc =

1. 1. LAlib = $(LAdir)/libf77blas.a $(LAdir)/libatlas.a

LAlib = $(LAdir)/libgoto.a -lm -L/usr/lib/gcc/i386-redhat-linux/4.1.2

...

CC = mpicc CCNOOPT = $(HPL_DEFS) CCFLAGS = $(HPL_DEFS) -O3

1. CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall

... LINKER = mpicc LINKFLAGS = $(CCFLAGS)

ARCHIVER = ar ARFLAGS = r RANLIB = echo

Now build this:-

[mpiuser@headnode hpl]$ make arch=Linux_PII_FBLAS_gm

... ... make[2]: Leaving directory `/cluster/mpiuser/hpl/testing/ptimer/Linux_PII_FBLAS_gm' ( cd testing/ptest/Linux_PII_FBLAS_gm; make ) make[2]: Entering directory `/cluster/mpiuser/hpl/testing/ptest/Linux_PII_FBLAS_gm' mpicc -DAdd_ -DF77_INTEGER=int -DStringSunStyle -I/cluster/mpiuser/hpl/include -I/cluster/mpiuser/hpl/include/Linux_PII_FBLAS_gm -O3 -o /cluster/mpiuser/hpl/bin/Linux_PII_FBLAS_gm/xhpl HPL_pddriver.o HPL_pdinfo.o HPL_pdtest.o /cluster/mpiuser/hpl/lib/Linux_PII_FBLAS_gm/libhpl.a /cluster/mpiuser/GotoBLAS/libgoto.a -lm -L/usr/lib/gcc/i386-redhat-linux/4.1.2 make /cluster/mpiuser/hpl/bin/Linux_PII_FBLAS_gm/HPL.dat make[3]: Entering directory `/cluster/mpiuser/hpl/testing/ptest/Linux_PII_FBLAS_gm' ( cp ../HPL.dat /cluster/mpiuser/hpl/bin/Linux_PII_FBLAS_gm ) make[3]: Leaving directory `/cluster/mpiuser/hpl/testing/ptest/Linux_PII_FBLAS_gm' touch dexe.grd make[2]: Leaving directory `/cluster/mpiuser/hpl/testing/ptest/Linux_PII_FBLAS_gm' make[1]: Leaving directory `/cluster/mpiuser/hpl' [mpiuser@headnode hpl]$

Now you should have HPL installed. We can check:-

[mpiuser@headnode hpl]$ ls /cluster/mpiuser/hpl/bin/Linux_PII_FBLAS_gm/ HPL.dat xhpl [mpiuser@headnode hpl]$

Alhumdulillah!

Now, Lets run LinPack:-

[mpiuser@headnode hpl]$ cd /cluster/mpiuser/hpl/bin/Linux_PII_FBLAS_gm/

[mpiuser@headnode Linux_PII_FBLAS_gm]$ cp HPL.dat HPL.dat.original

Here is the file, with various values inside it.:-

[mpiuser@headnode Linux_PII_FBLAS_gm]$ cat HPL.dat HPLinpack benchmark input file Innovative Computing Laboratory, University of Tennessee HPL.out output file name (if any) 6 device out (6=stdout,7=stderr,file) 4 # of problems sizes (N) 29 30 34 35 Ns 4 # of NBs 1 2 3 4 NBs 0 PMAP process mapping (0=Row-,1=Column-major) 3 # of process grids (P x Q) 2 1 4 Ps 2 4 1 Qs 16.0 threshold 3 # of panel fact 0 1 2 PFACTs (0=left, 1=Crout, 2=Right) 2 # of recursive stopping criterium 2 4 NBMINs (>= 1) 1 # of panels in recursion 2 NDIVs 3 # of recursive panel fact. 0 1 2 RFACTs (0=left, 1=Crout, 2=Right) 1 # of broadcast 0 BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM) 1 # of lookahead depth 0 DEPTHs (>=0) 2 SWAP (0=bin-exch,1=long,2=mix) 64 swapping threshold 0 L1 in (0=transposed,1=no-transposed) form 0 U in (0=transposed,1=no-transposed) form 1 Equilibration (0=no,1=yes) 8 memory alignment in double (> 0) [mpiuser@headnode Linux_PII_FBLAS_gm]$

Now edit the file HPL.dat and edit a few values:-

Remember, I have setup VMware machines, each with 1 x 2.2 GHz processor and 256 MB of memory. I will run the linpack test on a total of machine first.

The values to change in the file HPL.dat are changed based on following rules:-

First is P and Q. The rule is P x Q = total number of cores in the system, and Q >= P. So if you have 4 cores, then 2 * 2 = 4. I am benchmarking on two single core systems. So that is why I have 2. If you were doing a single node with two processor cores, then you could make P = 1 and Q = 2.

The second value you will need to change is the value of N. This is something you may experiment, before deciding on a final value. We can use the following formula as a good starting point:

Sqrt( .1 * (free -b of the available memory) * number of nodes)

ClusterVision tells us the formula as:-

(SquareRoot([GB/node]*1024*1024*128))*0.85

free -b is :-

[mpiuser@headnode Linux_PII_FBLAS_gm]$ free -b

            total       used       free     shared    buffers     cached

Mem: 261730304 208621568 53108736 0 29437952 122265600 -/+ buffers/cache: 56918016 204812288 Swap: 271425536 61440 271364096 [mpiuser@headnode Linux_PII_FBLAS_gm]$

value of free -b = 53108736

so lets start our basic calculator:-

[mpiuser@node1 Linux_PII_FBLAS_gm]$ bc bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. sqrt (.1 * (53108736)*2) 3259.1 quit

So we have 3259.1 as a result. But it should be a multiple of value of NB in HBL.dat. Since NB is 4 , as seen in the file below, and the suitable value for N would be 3256.

Lets boot two nodes:-

cd ~ [mpiuser@headnode ~]$ mpdboot -n 2 [mpiuser@headnode ~]$

[mpiuser@node1 ~]$ cd hpl/bin/Linux_PII_FBLAS_gm/

Try running the program on one node only, with the default HPL.dat:-

[mpiuser@node1 Linux_PII_FBLAS_gm]$ mpiexec -n 1 ./xhpl HPL ERROR from process # 0, on line 419 of function HPL_pdinfo: >>> Need at least 4 processes for these tests <<<

HPL ERROR from process # 0, on line 621 of function HPL_pdinfo: >>> Illegal input in file HPL.dat. Exiting ... <<<

[mpiuser@node1 Linux_PII_FBLAS_gm]$

Next, I changed the following lines in HPL.dat and run it on two nodes:- 3256 # of problems sizes (N) 4 # of NBs 1 # of problems sizes (N) 1 # of process grids (P x Q) 1 Ps 2 Qs

[mpiuser@headnode Linux_PII_FBLAS_gm]$ mpiexec -n 2 ./xhpl HPL ERROR from process # 0, on line 331 of function HPL_pdinfo: >>> Number of values of N is less than 1 or greater than 20 <<<

HPL ERROR from process # 0, on line 621 of function HPL_pdinfo: >>> Illegal input in file HPL.dat. Exiting ... <<<

[mpiuser@headnode Linux_PII_FBLAS_gm]$

Next, I changed the following lines in HPL.dat and run it on two nodes:- 4 # of problems sizes (N) 4 # of NBs 1 # of problems sizes (N) 1 # of process grids (P x Q) 1 Ps 2 Qs

[mpiuser@headnode Linux_PII_FBLAS_gm]$ mpiexec -n 2 ./xhpl

================================================================

HPLinpack 1.0a -- High-Performance Linpack benchmark -- January 20, 2004 Written by A. Petitet and R. Clint Whaley, Innovative Computing Labs., UTK

================================================================

An explanation of the input/output parameters follows: T/V : Wall time / encoded variant. N : The order of the coefficient matrix A. NB : The partitioning blocking factor. P : The number of process rows. Q : The number of process columns. Time : Time in seconds to solve the linear system. Gflops : Rate of execution for solving the linear system. ... ...

T/V N NB P Q Time Gflops

WR00R2C4 35 4 1 2 0.01 2.765e-03

||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0469732 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0515020 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0180039 ...... PASSED

================================================================

T/V N NB P Q Time Gflops

WR00R2R2 35 4 1 2 0.01 2.544e-03

||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0455498 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0499414 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0174583 ...... PASSED

================================================================

T/V N NB P Q Time Gflops

WR00R2R4 35 4 1 2 0.00 1.265e-02

||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0370092 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0405774 ...... PASSED ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0141849 ...... PASSED

================================================================

Finished 288 tests with the following results:

           288 tests completed and passed residual checks,
             0 tests completed and failed residual checks,
             0 tests skipped because of illegal input values.

End of Tests.

================================================================

[mpiuser@headnode Linux_PII_FBLAS_gm]$

So from the above output, I get 1.265e-02 Gflops, that means 0.01265 Gflops, which means, 12.65 MegaFlops.

The machines we have here has 1 processors. Each processor has 1 cores. Each core can do 4 Floating point operations per clock cycle. Each clock runs at a rate of 2.2GHz. Multiplying this out we can get the machines "Rpeak" or theoretical max performance:

1 processors * 1 cores * 4 FLOPS / clock cycle * 2.2 GHz = 8.8 GFlops

This is really confusing.

Q-1: How to find out a theoratical max performance value for a processor? Q-2: How to know how many FLOPS my processor can do in one clock cycle? Is it mentioned on some technical specification of a processor ? Q-3: How to correctly run Linpack?

Lets run the test again with a new HPL.dat file:-

[mpiuser@headnode Linux_PII_FBLAS_gm]$ cat HPL.dat HPLinpack benchmark input file Innovative Computing Laboratory, University of Tennessee HPL.out output file name (if any) 6 device out (6=stdout,7=stderr,file) 1 # of problems sizes (N) 3256 Ns 1 # of NBs 100 NBs 0 PMAP process mapping (0=Row-,1=Column-major) 1 # of process grids (P x Q) 1 Ps 2 Qs 16.0 threshold 3 # of panel fact 0 1 2 PFACTs (0=left, 1=Crout, 2=Right) 2 # of recursive stopping criterium 2 4 NBMINs (>= 1) 1 # of panels in recursion 2 NDIVs 3 # of recursive panel fact. 0 1 2 RFACTs (0=left, 1=Crout, 2=Right) 1 # of broadcast 0 BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM) 1 # of lookahead depth 0 DEPTHs (>=0) 2 SWAP (0=bin-exch,1=long,2=mix) 64 swapping threshold 0 L1 in (0=transposed,1=no-transposed) form 0 U in (0=transposed,1=no-transposed) form 1 Equilibration (0=no,1=yes) 8 memory alignment in double (> 0) [mpiuser@headnode Linux_PII_FBLAS_gm]$

[mpiuser@headnode Linux_PII_FBLAS_gm]$ mpiexec -n 2 ./xhpl > performance.txt [mpiuser@headnode Linux_PII_FBLAS_gm]$ grep ^WR performance.txt WR00L2L2 3256 100 1 2 5.06 4.552e+00 WR00L2L4 3256 100 1 2 4.14 5.563e+00 WR00L2C2 3256 100 1 2 4.88 4.715e+00 WR00L2C4 3256 100 1 2 4.65 4.950e+00 WR00L2R2 3256 100 1 2 4.94 4.661e+00 WR00L2R4 3256 100 1 2 4.61 4.994e+00 WR00C2L2 3256 100 1 2 5.60 4.114e+00 WR00C2L4 3256 100 1 2 3.80 6.056e+00 <<<--- Highest! WR00C2C2 3256 100 1 2 5.11 4.510e+00 WR00C2C4 3256 100 1 2 4.39 5.251e+00 WR00C2R2 3256 100 1 2 4.92 4.685e+00 WR00C2R4 3256 100 1 2 4.43 5.196e+00 WR00R2L2 3256 100 1 2 4.65 4.951e+00 WR00R2L4 3256 100 1 2 4.67 4.931e+00 WR00R2C2 3256 100 1 2 4.19 5.500e+00 WR00R2C4 3256 100 1 2 4.58 5.030e+00 WR00R2R2 3256 100 1 2 5.59 4.119e+00 WR00R2R4 3256 100 1 2 4.36 5.283e+00 [mpiuser@headnode Linux_PII_FBLAS_gm]$

As you can see above, one of the lines state 6.056 GFlops! Alhumdulillah.

Remember, this is the output from two node cluster.

Now lets see how much we should get in reality from one node:-

1 processors * 1 cores * 4 FLOPS / clock cycle * 2.2 GHz = 8.8 GFlops .

4Flops /clock Cycle is a fixed value, and is true for most of the modern processors. If I ignore any of the communication overheads between two nodes, I should be getting ideally , 8.8 GHz x 2 = 17.6 GFLOPS in total. Whereas, I am getting less than half (6 GFlops) on my test cluster. There is a reason to it. My nodes are VMware machines. As soon as they get a job, their CPU is shared in half. You also need to keep in mind the overhead/CPU usage of the host machine as well. So on real machines, I should get around 6GHz x 2 = 12 GFLOPS.

Efficiency of a cluster is simply : (Number of GFLOPS achieved / Number of theoratical GFLOPS) * 100 .

My cluster's efficiency looks like: (6 / 17.6) * 100 = 34 %

Later, I will show you the results from a real cluster, and the efficiency level. InshaAllah.

PXEClusterInstall

From WBITT's Cooker!

Revision as of 19:45, 14 February 2010

Contents

=

=

================================================================

================================================================

================================================================

================================================================

================================================================

================================================================

Views

Personal tools

Navigation

Search

Toolbox

@@ Line 584: / Line 584: @@
 Note: This excercise assumes that RSA HOST Keys were  scanned for each host and saved in /etc/ssh/ssh_known_hosts . However, it doesn't make sense. You see, in the beginning there will be only one host RSA key in the /etc/ssh/ssh_known_hosts on the headnode. And that would be the entry for headnode only. And we also see that we could not know the RSA host key of any nodes which are not installed yet. So, one way to do it is to make this ssh_known_hosts file available to the node as well over http. We will copy this file from the headnode to the client in the %post of client kickstart.
-But, how would the headnode know that it is time that a particular node is done installation and it can generate it's ssh-host-key and add it to its /etc/ssh/ssh_known_hosts ?  I wonder how aramco is doing it ?
+But, how would the headnode know that it is time that a particular node is done installation and it can generate it's ssh-host-key and add it to its /etc/ssh/ssh_known_hosts ?  I wonder how my company is doing it ?
 Remember we are doing this manually anyway. I mean the node restart one by one . We are not doing it through any management software as yet. So we need to manually add the ssh-host-key of each node to the /etc/ssh/ssh_known_hosts file on the server, at the end of installation of each node.
@@ Line 1,400: / Line 1,400: @@
 [mpiuser@headnode Linux_PII_FBLAS_gm]$ mpiexec -n 2 ./xhpl
 ============================================================================
 HPLinpack 1.0a  --  High-Performance Linpack benchmark  --   January 20, 2004