PXEBooting

From WBITT's Cooker!

(Difference between revisions)

Revision as of 11:46, 6 February 2010

PXE boot-howto-simplified:

=============

Scenario: storage.homedomain.com : Install server kvm.homedomain.com : Physical host to host kvm virtual machines. Will be used as test server to be installed using PXE from the install server. xen.homedomain.com : Physical host to host xen virtual machines. Will be used as test server to be installed using PXE from the install server.

The install server (storage):

Copy the (expanded) CentOS-5.4-x86_64 DVD in a directory /data/cdimages/CentOS-5.4-x86_64 .

Import the gpg key. It would be wise to do it at this moment:

rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

Create a local repository for this server, so it can install any required packages , even if it is not connected to internet.

[root@storage ~]# vi /etc/yum.repos.d/CentOS-localmedia.repo
[CentOS-5.4-x86_64-local]
name=RedHat Enterprise Linux $releasever - $basearch
baseurl=file:///data/cdimages/CentOS-5.4-x86_64/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[root@storage ~]#

You can verify the new repository using:

[root@storage ~]# yum repolist
Loaded plugins: fastestmirror
Determining fastest mirrors
CentOS-5.4-x86_64-local                                                                     | 2.1 kB     00:00
CentOS-5.4-x86_64-local/primary_db                                                          | 2.0 MB     00:00
repo id                                     repo name                                               status
CentOS-5.4-x86_64-local                      RedHat Enterprise Linux 5 - x86_64                      enabled: 3,348
repolist: 3,348
[root@storage ~]#

You must verify it using the following method too:-

[root@storage ~]# yum list | tail
yum-protect-packages.noarch          1.1.16-13.el5.centos CentOS-5.4-x86_64-local
yum-protectbase.noarch               1.1.16-13.el5.centos CentOS-5.4-x86_64-local
yum-security.noarch                  1.1.16-13.el5.centos CentOS-5.4-x86_64-local
yum-tmprepo.noarch                   1.1.16-13.el5.centos CentOS-5.4-x86_64-local
yum-updateonboot.noarch              1.1.16-13.el5.centos CentOS-5.4-x86_64-local
yum-utils.noarch                     1.1.16-13.el5.centos CentOS-5.4-x86_64-local
yum-verify.noarch                    1.1.16-13.el5.centos CentOS-5.4-x86_64-local
yum-versionlock.noarch               1.1.16-13.el5.centos CentOS-5.4-x86_64-local
zenity.x86_64                        2.16.0-2.el5         CentOS-5.4-x86_64-local
zsh-html.x86_64                      4.2.6-3.el5          CentOS-5.4-x86_64-local
[root@storage ~]#

This shows that the repository and it's packages is being correctly loaded / listed.

You need to have the following installed on the install server:

dhcp (server) syslinux tfpt httpd

[root@storage ~]# yum -y install httpd tftp-server syslinux dhcp

Now create an httpd alias , which will be used by the machines being installed, to pull pakcages from httpd repository.

vi /etc/httpd/conf.d/CentOS-5.4-x86_64.conf

Alias /CentOS-5.4-x86_64 /data/cdimages/CentOS-5.4-x86_64/
<Location /CentOS-5.4-x86_64>
    Order deny,allow
    Allow from all
    Options +Indexes
</Location>

The "Options +Indexes" seems to be un-necessary, but it is better to put it here. This will help you later to download individual packages from this local repository in difficult situations.

Restart Apache server:

service httpd restart

And check through a browser, if you are able to view the web page of your repository.

If you don't have a web browser available to you, try using a simple wget to pull index.html from this link.

[root@storage ~]# wget http://192.168.1.100/CentOS-5.4-x86_64/
--2010-02-05 11:41:11--  http://192.168.1.100/CentOS-5.4-x86_64/
Connecting to 192.168.1.100:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6623 (6.5K) [text/html]
Saving to: `index.html'

100%[========================================================================>] 6,623       --.-K/s   in 0s

2010-02-05 11:41:11 (134 MB/s) - `index.html' saved [6623/6623]

[root@storage ~]#

Time to setup dhcp service on this (install) server:

# vi /etc/dhcpd.conf
ddns-update-style interim;
ignore client-updates;


subnet 192.168.1.0 netmask 255.255.255.0 {

# --- default gateway
        option routers                  192.168.1.1;
        option subnet-mask              255.255.255.0;
        option domain-name              "homedomain.com";
        filename "pxelinux.0";
        range dynamic-bootp 192.168.1.11 192.168.1.20;
        default-lease-time 21600;
        max-lease-time 43200;
        next-server 192.168.1.100;

}

Start the dhcpd service. Make sure that there is no other DHCP service running elsewhere on your network.

service dhcpd restart

Now it is time to setup the TFTP service:-

[root@storage ~]# cat /etc/xinetd.d/tftp
service tftp
{
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /tftpboot
        disable                 = no
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}
[root@storage ~]#

Restart the xinetd service:-

service xinetd restart
chkconfig --level 35 xinetd on

Next, we need to copy two special files from a special location of distribution media/repository to /tftpboot directory.

[root@storage ~]# ls /data/cdimages/CentOS-5.4-x86_64/images/pxeboot
boot.iso  diskboot.img  minstg2.img  pxeboot  README  stage2.img  TRANS.TBL  xen
[root@storage ~]#

We will copy the vmlinuz and initrd.img files, from this location to /tftpboot directory :-

[root@storage ~]# cp /data/cdimages/CentOS-5.4-x86_64/images/pxeboot/* /tftpboot/

[root@storage ~]# ls /tftpboot/ -l
total 9140
-rw-r--r-- 1 root root 7397850 Feb  5 12:41 initrd.img
-rw-r--r-- 1 root root     265 Feb  5 12:41 README
-r--r--r-- 1 root root     659 Feb  5 12:41 TRANS.TBL
-rw-r--r-- 1 root root 1932284 Feb  5 12:41 vmlinuz
[root@storage ~]#

The README and TRANS.TBL are not needed, but there is no harm in having them here in /tftpboot/ .

Basically we need only vmlinuz and initrd.img .

Now is the time to copy the pxelinux.0 file (from the syslinux package) to /tftpboot directory .

# cp /usr/lib/syslinux/pxelinux.0    /tftpboot/

Now make sure that all files and direcoties inside /tftpboot is world readable.

# chmod +r /tftpboot/* -R

[root@storage tftpboot]# ls -l
total 9152
-rw-r--r-- 1 root root 7397850 Feb  5 12:41 initrd.img
-rw-r--r-- 1 root root   13148 Feb  5 12:53 pxelinux.0
drwxr-xr-x 2 root root    4096 Feb  5 12:48 pxelinux.cfg
-rw-r--r-- 1 root root 1932284 Feb  5 12:41 vmlinuz
[root@storage tftpboot]#

(I have removed the README and TRANS.TBL files from /tftpboot to remove clutter).

PXE configuration detail:- When a client boots, by default it will look for a configuration file from TFTP, with the same name as it's MAC address. However, afrer trying several options, it will fall back to requesting a default file, with the name "default". This file needs to be in a directory in /tftp of the install server.

</pre>

mkdir /tftpboot/pxelinux.cfg

vi /tftpboot/pxelinux.cfg/default

prompt 1 timeout 5 default linux label linux kernel vmlinuz append vga=normal initrd=initrd.img </pre>

Once these settings are done, you can now try rebooting your client computer and see if it is able to get this TFTP image from this install server.

You will need to enable PXE boot from the network card in the BIOS of that machine. (This is the defult/intended behavior in an HPCC environment).

Note: If you get TFTP open timeouts on the client machine, (the machine, which is to be installed through PXE boot), then may be you did not start your xinetd service yet. Or may be a firewall issue.

I had my client machine setup to boot from PXE, and here is what I see in /var/log/messages:

Feb  5 12:59:54 storage dhcpd: DHCPDISCOVER from 00:13:72:81:3a:3d via eth0
Feb  5 12:59:55 storage dhcpd: DHCPOFFER on 192.168.1.20 to 00:13:72:81:3a:3d via eth0
Feb  5 12:59:58 storage dhcpd: DHCPREQUEST for 192.168.1.20 (192.168.1.100) from 00:13:72:81:3a:3d via eth0
Feb  5 12:59:58 storage dhcpd: DHCPACK on 192.168.1.20 to 00:13:72:81:3a:3d via eth0
Feb  5 12:59:58 storage in.tftpd[19447]: tftp: client does not accept options

Note: The message "in.tftpd[19447]: tftp: client does not accept options" , is not an error message. Just information, nothing to worry about.

By doing this, you have managed to start up the interactive installation . Congratulations!

Please note, that this is not what we want. Read the next section.

=================================================================================

We want nodes to install through kickstart, automatically. And also, we do not want them to get stuck in an install loop for ever. Means, that we want that a node, should only get installed, when it is asked to, and should boot from local disk, when the installation is over.

We will achieve our objectives in steps. First we fix the automatic installation problem, using kickstart.

By default, when a system is installed, there is a file in /root , named anaconda-ks.cfg . We will use the same file as tempelate. Here:

cp /root/anaconda-ks.cfg /var/www/html/compute-ks.cfg

[root@storage ~]# vi /var/www/html/compute-ks.cfg
install
text
url --url http://192.168.1.100/CentOS-5.4-x86_64
lang en_US.UTF-8
keyboard us
network --device eth0 --bootproto dhcp
rootpw --iscrypted $1$VQPyk3Ev$JePfY50WaA.aBhKT3xsBq.
firewall --disabled
authconfig --enableshadow --enablemd5
selinux --disabled
timezone Asia/Riyadh
bootloader --location=mbr --driveorder=sda --append=""
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
zerombr
clearpart --all --initlabel
# part / --fstype ext3 --size=1 --grow
# part swap --size=256
part / --fstype ext3 --size=3000
part swap --size=512
reboot

%packages
@core

Make sure that this file is readable by Apache, or world readable.

[root@storage ~]# ls -l /var/www/html/
total 8
-rw------- 1 root root 766 Feb  5 13:14 compute-ks.cfg
-rw-r--r-- 1 root root  31 Nov 23 07:31 index.html

[root@storage ~]# chmod +r /var/www/html/*.cfg

As you can see that all compute node, as many as ther would be, would install without a hostname, and without a permanent IP assigned to them. Also as soon as they get installed, they will reboot and go through another install cycle. Which is not desired. However, before we go on and solve that issue, lets try to install the node as per above configuration. You will need to modify the /tftpboot/pxelinux.cfg/default file as :-

Edit the tftpboot file again and add extra options:-

[root@headnode ~]# vi /tftpboot/pxelinux.cfg/default

prompt 1
timeout 5
default linux
label  linux
        kernel vmlinuz
        append vga=normal initrd=initrd.img ip=dhcp ksdevice=eth0 ks=http://192.168.1.100/compute-ks.cfg

Reboot your client machine and see the magic. You should be able to see activity in the apache access_log. Entries such as following:-

[root@storage ~]# tail -f /var/log/httpd/access_log
. . . 
. . . 
192.168.1.20 - - [05/Feb/2010:13:22:41 +0300] "GET /CentOS-5.4-x86_64/CentOS/pcsc-lite-libs-1.4.4-0.1.el5.x86_64.rpm HTTP/1.1" 200 24120 "-" "urlgrabber/3.1.0 yum/3.2.22"
. . .
. . .

During this install session, I was able to see the following as the last entries in the apache access_log :-

192.168.1.20 - - [05/Feb/2010:13:24:54 +0300] "GET /CentOS-5.4-x86_64/CentOS/NetworkManager-0.7.0-9.el5.x86_64.rpm HTTP/1.1" 200 1099937 "-" "urlgrabber/3.1.0 yum/3.2.22"
192.168.1.20 - - [05/Feb/2010:13:24:54 +0300] "GET /CentOS-5.4-x86_64/CentOS/NetworkManager-glib-0.7.0-9.el5.i386.rpm HTTP/1.1" 200 83448 "-" "urlgrabber/3.1.0 yum/3.2.22"

As you might guess, these entries may vary as per the installation packages selected. In other words, it is not guaranteed that NetworkManager would be the last package obtained from the repository during an install.

Basically I am trying to tell you that we need to find/device a way so we know that when installation of a particular node is complete.

End of day.

Here are things to do / ideas: Create a text file of all hostnames, IP and MAC addresses, delimited by a space. Create a program perl/bash, which will read this file and generates a proper kickstart and related tftpboot file in /tftpboot/pxelinux.cfg/ directory. Create a small text file and make it available in the /var/www/html directory, which will be pulled and copied to the installed node in /tmp, during (at the end of) the %post. Some program should constantly monitor the apache logs and check if a node accesses this file. When some node accesses this file, that means that the node is now completed installation and it's tftpboot file can be removed. This also means that the default tftpboot file should only contain a boot from local disk option. This way if some nodes mac-based tftp file is removed, its PXE will fall through and land on the default, which will make it boot from the local disk. This should do it.

Will work on this next day.

======================================================

Day 2:-

I created a file /tftpboot/pxelinux.cfg/00-13-72-81-3a-3d , with the following contents:

[root@storage pxelinux.cfg]# cat 00-13-72-81-3a-3d
prompt 1
timeout 10
default linux
label  linux
        kernel vmlinuz
        append vga=normal initrd=initrd.img ip=dhcp ksdevice=eth0 ks=http://192.168.1.100/compute-ks.cfg

[root@storage pxelinux.cfg]#

The default file is modified to look like this:

[root@storage pxelinux.cfg]# cat default
prompt 1
timeout 10
default localdisk
label localdisk
    localboot 0

However, when I rebooted the client, the TFTP client did was infact searching for a file named "01-00-13-72-81-3a-3d" . This was strange. Thus, I renamed the file 00-13-72-81-3a-3d to 01-00-13-72-81-3a-3d. Rebooted the client and it installed correctly.

Note: The "01", before the MAC address represents a hardware type of ethernet.

As soon as I saw that it is about to finish, I renamed the PXE boot filename so the TFTP boot client will not load the mac-based file and instead fall through to load the default file.

[root@storage pxelinux.cfg]# mv 01-00-13-72-81-3a-3d 01-00-13-72-81-3a-3d.orig

Created a html file /var/www/html/lastpackage.html: ( Actually it doesn't have to be an html file )

[root@storage html]# cat lastpackage.html
This is the last file, which is downloaded from the client, during installation. When it is downloaded, it means that the installation is complete and now the mac-based tftp installation file can be removed from /tftpboot/pxelinux.cfg/.
[root@storage html]#

I see the following entry in the apache access_log file when the node is done installation.

[root@storage html]# tail -f /var/log/httpd/access_log
. . . 
. . . 
192.168.1.20 - - [06/Feb/2010:02:52:33 +0300] "GET /CentOS-5.4-x86_64/CentOS/NetworkManager-0.7.0-9.el5.x86_64.rpm HTTP/1.1" 200 1099937 "-" "urlgrabber/3.1.0 yum/3.2.22"
192.168.1.20 - - [06/Feb/2010:02:52:34 +0300] "GET /CentOS-5.4-x86_64/CentOS/NetworkManager-glib-0.7.0-9.el5.i386.rpm HTTP/1.1" 200 83448 "-" "urlgrabber/3.1.0 yum/3.2.22"
192.168.1.20 - - [06/Feb/2010:02:52:36 +0300] "GET /lastpackage.html HTTP/1.0" 200 240 "-" "Wget/1.11.4 Red Hat modified"
. . . 
192.168.1.20 - - [06/Feb/2010:02:52:36 +0300] "GET /lastpackage.html HTTP/1.0" 200 240 "-" "Wget/1.11.4 Red Hat modified"

If I can now make a small program, which could monitor this log file and remove the mac-based tftp file as soon as it sees this entry, it would be superb.

First, I need to have a translation file, having IP, hostname and MAC information in it.

[root@storage ~]# vi /root/host-ip-mac.txt
# Format of this file is :
# hosntname IP MAC
# hostname a.b.c.d aa-bb-cc-dd-ee-ff
kvm     192.168.1.20    00-13-72-81-3a-3d


[root@storage ~]# egrep -v "#|^$"  /root/host-ip-mac.txt
kvm     192.168.1.20    00-13-72-81-3a-3d
[root@storage ~]#

Here is a bit of demo, how can I select different columns:

[root@storage ~]# egrep -v "#|^$"  /root/host-ip-mac.txt  |awk '{print $1}'
kvm

[root@storage ~]# egrep -v "#|^$"  /root/host-ip-mac.txt  |awk '{print $2}'
192.168.1.20

[root@storage ~]# egrep -v "#|^$"  /root/host-ip-mac.txt  |awk '{print $3}'
00-13-72-81-3a-3d

Now I needed a log file monitoring tool, which I can use to monitor the apache log file and take an action when a match is found. I found few software with announced capability of what I was looking for. These are logtail, log-guardian, swatch.

I find log-guardian to be complicated. Just to be complete, I am showing below, what I experienced.

[root@storage ~]# wget http://www.tifaware.com/code/log-guardian/log-guardian -P /root

chmod +x /root/log-guardian

[root@storage ~]# ./log-guardian
Can't locate File/Tail.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8) at ./log-guardian line 144.
BEGIN failed--compilation aborted at ./log-guardian line 144.
[root@storage ~]#


Log Guardian needs Tail.pm .

[root@storage ~]# wget http://search.cpan.org/CPAN/authors/id/M/MG/MGRABNAR/File-Tail-0.99.3.tar.gz -P /root/     


[root@storage ~]# tar xzf File-Tail-0.99.3.tar.gz

[root@storage ~]# cd File-Tail-0.99.3

[root@storage File-Tail-0.99.3]# perl Makefile.PL

[root@storage File-Tail-0.99.3]# make

[root@storage File-Tail-0.99.3]# make install
Installing /usr/lib/perl5/site_perl/5.8.8/File/Tail.pm
Installing /usr/share/man/man3/File::Tail.3pm
Writing /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/File/Tail/.packlist
Appending installation info to /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/perllocal.pod
[root@storage File-Tail-0.99.3]#

Swatch seems to be the other alternative:- (rpmforge repository) or direct download.

Help: http://linsec.ca/Using_swatch_to_Monitor_Logfiles

wget http://sourceforge.net/projects/swatch/files/swatch/3.2.3/swatch-3.2.3.tar.gz/download

[root@storage swatch-3.2.3]# perl Makefile.PL
Checking if your kit is complete...
Looks good
Warning: prerequisite Date::Calc 0 not found.
Warning: prerequisite Date::Format 0 not found.
Warning: prerequisite Date::Manip 0 not found.
Writing Makefile for swatch
[root@storage swatch-3.2.3]#

[root@storage swatch-3.2.3]# yum -y install perl-Date-Calc perl-DateManip perl-HTML-Parser perl-TimeDate perl-XML-Parser


[root@storage swatch-3.2.3]# perl Makefile.PL
Writing Makefile for swatch
[root@storage swatch-3.2.3]#


make
make install


[root@storage ~]# cat /usr/local/etc/swatch.conf
watchfor   /lastpackage.html /
        echo

[root@storage ~]#


[root@storage ~]# swatch --config-file=/usr/local/etc/swatch.conf --tail-file=/tmp/swatch.log --script-dir=/usr/local/swatch/

I created a script, which would take in an IP and remove the corresponding tftp file .

[root@storage swatch]# cat /usr/local/swatch/remove-tftp-file.sh
#!/bin/bash
MAPFILE=/root/host-ip-mac.txt
IP=$1
echo "Recieved IP is: ${IP}"
MAC=$(grep ${IP} ${MAPFILE} | awk '{print $3}' )
TFTPMACFILE="01-${MAC}"
echo "Executing: rm -f /tftpboot/pxelinux.cfg/${TFTPMACFILE}"
[root@storage swatch]#

End of day 2.

PXEBooting

From WBITT's Cooker!

Revision as of 11:46, 6 February 2010

=============

=================================================================================

======================================================

Views

Personal tools

Navigation

Search

Toolbox