CobblerTest

From WBITT's Cooker!

(Difference between revisions)
Jump to: navigation, search
(→==)
m (→Define Systems)
 
(17 intermediate revisions not shown)
Line 1: Line 1:
-
Cobbler Howto:-
+
= Cobbler Howto =
-
==================
+
= Cobbler install on Dell 1950 Servers =
 +
This setup consists of four Dell PowerEdge 1950 servers, connected to each other in a cluster form. The intended purpose of this excercise is to find a way to provision compute nodes, without hassle. This should be ideal for data centers (both HPCC and non-HPCC).
-
help from:=
 
-
http://www.openskill.info/blog.php?ID=16
+
Help from:-
-
http://magazine.redhat.com/2007/08/10/cobbler-how-to-set-up-a-network-boot-server-in-10-minutes/
+
-
http://www.enterprisenetworkingplanet.com/netsysm/article.php/3773596
+
-
http://www.kanarip.com/courses/Deploying_Linux_Using_Cobbler/
+
 +
* http://www.openskill.info/blog.php?ID=16
 +
* http://magazine.redhat.com/2007/08/10/cobbler-how-to-set-up-a-network-boot-server-in-10-minutes/
 +
* http://www.enterprisenetworkingplanet.com/netsysm/article.php/3773596
 +
* http://www.kanarip.com/courses/Deploying_Linux_Using_Cobbler/
-
Centos 5.4 x86_64
 
 +
The system has Centos 5.4 x86_64 installed as base OS. You might want to update it before continuing.
 +
 +
<pre>
yum -y  update
yum -y  update
 +
</pre>
 +
 +
 +
After this you should make sure that you have the following installed, from your distribution media / repository.
 +
 +
* httpd
 +
* tftp-server
 +
* xinetd
 +
* reposync (yum-utils provides reposync)
 +
 +
<pre>
 +
yum -y install httpd yum-utils tftp-server xinetd dhcp
 +
</pre>
 +
Make sure you are not running kernel-xen on your base OS. (Although it does not matter, but I just want to keep things simple). My scenario requires me not to run kernel-xen. (We are not using virtualization features. These are pure compute nodes.)
Using the link: https://fedorahosted.org/cobbler/wiki/DownloadInstructions  
Using the link: https://fedorahosted.org/cobbler/wiki/DownloadInstructions  
-
Setup Cobbler Repositories:-
+
Setup Cobbler Repositories:- (Needs internet connection).
-
[root@test-plccg ~]# vi /etc/yum.repos.d/CentOS-Base.repo  
+
<pre>
 +
[root@headnode ~]# vi /etc/yum.repos.d/CentOS-Base.repo  
. . .
. . .
. . .  
. . .  
Line 32: Line 50:
. . .
. . .
. . .   
. . .   
-
[root@test-plccg ~]#  
+
[root@headnode ~]#  
 +
</pre>
-
Install cobbler:-
+
Now, install cobbler:-
-
[root@test-plccg ~]# yum -y install cobbler koan
+
[root@headnode ~]# yum -y install cobbler koan cman
-
[root@test-plccg cdimages]# ls
+
 
 +
I have various CD images in my /data/cdimages location. I expanded the ISO image and copied it in CentOS-5.4-x86_64 .
 +
 
 +
<pre>
 +
[root@headnode cdimages]# ls
CentOS-5.3-i386.DVD.iso         
CentOS-5.3-i386.DVD.iso         
CentOS-5.4-x86_64               
CentOS-5.4-x86_64               
CentOS-5.4-x86_64-bin-DVD.iso   
CentOS-5.4-x86_64-bin-DVD.iso   
-
[root@test-plccg cdimages]#
+
[root@headnode cdimages]#
-
 
+
</pre>
-
--------------------
+
-
 
+
-
service httpd restart
+
-
 
+
-
http://10.1.112.121/cobbler
+
-
 
+
-
 
+
-
--------------------
+
-
 
+
-
 
+
-
Import repository:-
+
 +
Assuming you have entire CD media copied in your /data/cdimages/CentOS-5.4-x86_64  , you are now ready to import the repository.
 +
<pre>
cobbler import --mirror=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64
cobbler import --mirror=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64
 +
</pre>
 +
or
or
-
cobbler import --path=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64
 
 +
<pre>
 +
cobbler import --path=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64
 +
</pre>
 +
During import you should get some screen out put as:
                      
                      
-
[root@test-plccg cdimages]# cobbler import --mirror=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64
+
<pre>
 +
[root@headnode cdimages]# cobbler import --mirror=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64
. . .
. . .
  . . .
  . . .
Line 112: Line 132:
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/xen
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/xen
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot
-
[root@test-plccg cdimages]#
+
[root@headnode cdimages]#
 +
</pre>
-
Install the following using yum :-
 
-
yum -y install reposync yum-utils cman
+
Time for cobbler configurations:-
 +
<pre>
 +
[root@headnode ~]# vi /etc/cobbler/settings
 +
server = 192.168.1.203
 +
nextserver=192.168.1.203
 +
pxe_just_once = 1
 +
. . .
 +
</pre>
-
Time for some configurations:-
+
Now you should start the cobblerd service along httpd, dhcpd and tftp server.
-
vi /etc/cobbler/settings
 
-
 
+
<pre>
-
(some settings)
+
chkconfig --level 35 cobblerd on
 +
chkconfig --level 35 httpd on
 +
chkconfig --level 35 xinetd on
 +
chkconfig --level 35 dhcpd on
service cobblerd restart  
service cobblerd restart  
 +
service httpd restart
 +
service dhcpd restart
 +
service xinetd restart
 +
</pre>
-
chkconfig --level 35 cobblerd on
+
Should enable the tftp service in /etc/xinetd.d/tftp , and change disabled=no .
 +
 
 +
Now run cobbler check and fix any obvious errors.
 +
<pre>
cobbler check # important
cobbler check # important
 +
</pre>
 +
You may want to disable un-necessary services on this server before moving on:
 +
 +
<pre>
 +
chkconfig --level 35  NetworkManager  off                                                         
 +
chkconfig --level 35  autofs  off                                                                 
 +
chkconfig --level 35  avahi-daemon  off                                                           
 +
chkconfig --level 35  avahi-dnsconfd  off                                                         
 +
chkconfig --level 35  bluetooth  off                                                             
 +
chkconfig --level 35  cups  off                                                                   
 +
chkconfig --level 35  dc_client  off                                                             
 +
chkconfig --level 35  dc_server  off                                                             
 +
chkconfig --level 35  dhcrelay  off                                                               
 +
chkconfig --level 35  dnsmasq  off                                                               
 +
chkconfig --level 35  dund  off                                                                   
 +
chkconfig --level 35  firstboot  off                                                             
 +
chkconfig --level 35  gpm  off                                                                   
 +
chkconfig --level 35  hidd  off                                                                   
 +
chkconfig --level 35  ip6tables  off                                                             
 +
chkconfig --level 35  iptables  off                                                               
 +
chkconfig --level 35  irda  off                                                                   
 +
chkconfig --level 35  iscsi  off                                                                 
 +
chkconfig --level 35  iscsid  off                                                                 
 +
chkconfig --level 35  kdump  off                                                                 
 +
chkconfig --level 35  libvirtd  off                                                               
 +
chkconfig --level 35  lisa  off                                                                   
 +
chkconfig --level 35  lvm2-monitor  off                                                           
 +
chkconfig --level 35  mdmonitor  off                                                             
 +
chkconfig --level 35  mdmpd  off                                                                 
 +
chkconfig --level 35  multipathd  off                                                             
 +
chkconfig --level 35  mysqld  off                                                                 
 +
chkconfig --level 35  named  off                                                                 
 +
chkconfig --level 35  netconsole  off                                                             
 +
chkconfig --level 35  netfs  off                                                                 
 +
chkconfig --level 35  netplugd  off                                                               
 +
chkconfig --level 35  nfs  off                                                                   
 +
chkconfig --level 35  nfslock  off                                                               
 +
chkconfig --level 35  nscd  off                                                                   
 +
chkconfig --level 35  ntpd  off                                                                   
 +
chkconfig --level 35  oddjobd  off                                                               
 +
chkconfig --level 35  pand  off                                                                   
 +
chkconfig --level 35  pcscd  off
 +
chkconfig --level 35  psacct  off
 +
chkconfig --level 35  qemu  off
 +
chkconfig --level 35  rawdevices  off
 +
chkconfig --level 35  rdisc  off
 +
chkconfig --level 35  readahead_later  off
 +
chkconfig --level 35  restorecond  off
 +
chkconfig --level 35  rpcgssd  off
 +
chkconfig --level 35  rpcidmapd  off
 +
chkconfig --level 35  rpcsvcgssd  off
 +
chkconfig --level 35  saslauthd  off
 +
chkconfig --level 35  sendmail  off
 +
chkconfig --level 35  setroubleshoot  off
 +
chkconfig --level 35  smartd  off
 +
chkconfig --level 35  smb  off
 +
chkconfig --level 35  squid  off
 +
chkconfig --level 35  tcsd  off
 +
chkconfig --level 35  tux  off
 +
chkconfig --level 35  vncserver  off
 +
chkconfig --level 35  wdaemon  off
 +
chkconfig --level 35  winbind  off
 +
chkconfig --level 35  wpa_supplicant  off
 +
chkconfig --level 35  xend  off
 +
chkconfig --level 35  xendomains  off
 +
chkconfig --level 35  ypbind  off
 +
chkconfig --level 35  yum-updatesd  off
 +
</pre>
End of day 1.
End of day 1.
-
=========================
+
= Day 2 =
-
Day 2:
+
<pre>
-
Cobbler howto-2
+
[root@headnode ~]# cobbler check
-
=============
+
-
 
+
-
 
+
-
[root@test-plccg ~]# cobbler check
+
The following potential problems were detected:
The following potential problems were detected:
#0: service dhcpd is not running
#0: service dhcpd is not running
Line 151: Line 251:
#2: yum-utils need to be at least version 1.1.17 for reposync -l, current version is 1.1.16
#2: yum-utils need to be at least version 1.1.17 for reposync -l, current version is 1.1.16
#3: The default password used by the sample templates for newly installed machines (default_password_crypted in /etc/cobbler/settings) is still set to 'cobbler' and should be changed
#3: The default password used by the sample templates for newly installed machines (default_password_crypted in /etc/cobbler/settings) is still set to 'cobbler' and should be changed
-
[root@test-plccg ~]#  
+
[root@headnode ~]#  
-
 
+
</pre>
 +
As you can see, 1-3 are more important to check. 4 is less important.
-
[root@test-plccg ~]# cobbler list
 
-
distro Centos5.4-x86_64
 
-
  profile Centos5.4-x86_64
 
-
  profile rescue-Centos5.4-x86_64
 
-
distro Centos5.4-xen-x86_64
 
-
  profile Centos5.4-xen-x86_64
 
-
[root@test-plccg ~]#
 
 +
Check various lists from cobbler. These are created by default when you import the repository for the first time.
   
   
-
[root@test-plccg ~]# cobbler distro list
+
<pre>
 +
[root@headnode ~]# cobbler distro list
Centos5.4-x86_64
Centos5.4-x86_64
Centos5.4-xen-x86_64
Centos5.4-xen-x86_64
-
[root@test-plccg ~]#
+
[root@headnode ~]#
-
[root@test-plccg ~]# cobbler profile list
+
[root@headnode ~]# cobbler profile list
Centos5.4-x86_64
Centos5.4-x86_64
Centos5.4-xen-x86_64
Centos5.4-xen-x86_64
rescue-Centos5.4-x86_64
rescue-Centos5.4-x86_64
-
[root@test-plccg ~]#  
+
[root@headnode ~]#  
 +
</pre>
-
No systems are registered with cobbler at the moment:-
+
Currently. No systems are registered with cobbler at the moment. As you can see below:
-
[root@test-plccg ~]# cobbler system list
+
<pre>
-
[root@test-plccg ~]#
+
[root@headnode ~]# cobbler system list
 +
[root@headnode ~]#
 +
</pre>
-
Lets create a profile for compute-nodes:-
+
First, create a (separate) profile for compute-nodes:-
-
[root@test-plccg ~]# cobbler profile copy --name=Centos5.4-x86_64 --newname=ComputeNodes
+
<pre>
 +
[root@headnode ~]# cobbler profile copy --name=Centos5.4-x86_64 --newname=ComputeNodes
-
[root@test-plccg ~]# cobbler profile list
+
[root@headnode ~]# cobbler profile list
Centos5.4-x86_64
Centos5.4-x86_64
Centos5.4-xen-x86_64
Centos5.4-xen-x86_64
ComputeNodes                # <------------ Newly added profile!
ComputeNodes                # <------------ Newly added profile!
rescue-Centos5.4-x86_64
rescue-Centos5.4-x86_64
-
[root@test-plccg ~]#
+
[root@headnode ~]#
-
[root@test-plccg ~]# cobbler profile report --name=ComputeNodes
+
[root@headnode ~]# cobbler profile report --name=ComputeNodes
profile              : ComputeNodes
profile              : ComputeNodes
distro              : Centos5.4-x86_64
distro              : Centos5.4-x86_64
Line 224: Line 324:
virt type            : qemu
virt type            : qemu
-
[root@test-plccg ~]#  
+
[root@headnode ~]#  
 +
</pre>
 +
The name of default.ks needs to be changed, or fill default.ks with your kickstart, because it is empty by default.
-
[root@test-plccg ~]# cat /var/lib/cobbler/kickstarts/default.ks
+
<pre>
 +
[root@headnode ~]# cat /var/lib/cobbler/kickstarts/default.ks
# this file intentionally left blank
# this file intentionally left blank
# admins:  edit it as you like, or leave it blank for non-interactive install
# admins:  edit it as you like, or leave it blank for non-interactive install
 +
</pre>
 +
There is a sample.ks, which we will use for our compute nodes.
-
[root@test-plccg ~]# cat /var/lib/cobbler/kickstarts/sample.ks
+
<pre>
 +
[root@headnode ~]# cat /var/lib/cobbler/kickstarts/sample.ks
#platform=x86, AMD64, or Intel EM64T
#platform=x86, AMD64, or Intel EM64T
# System authorization information
# System authorization information
Line 301: Line 407:
$kickstart_done
$kickstart_done
# End final steps
# End final steps
-
[root@test-plccg ~]#
+
[root@headnode ~]#
 +
</pre>
 +
Copy the sample file as Compute.ks :
-
[root@test-plccg ~]# cd /var/lib/cobbler/kickstarts/
+
<pre>
 +
[root@headnode ~]# cd /var/lib/cobbler/kickstarts/
-
[root@test-plccg kickstarts]# cp sample.ks Compute.ks
+
[root@headnode kickstarts]# cp sample.ks Compute.ks
 +
</pre>
-
[root@test-plccg kickstarts]# vi Compute.ks
+
Edit the kickstart file Compute.ks and make necessary changes:
 +
 
 +
<pre>
 +
[root@headnode kickstarts]# vi Compute.ks
#platform=x86, AMD64, or Intel EM64T
#platform=x86, AMD64, or Intel EM64T
# System authorization information
# System authorization information
Line 383: Line 496:
$kickstart_done
$kickstart_done
# End final steps
# End final steps
-
[root@test-plccg kickstarts]#  
+
[root@headnode kickstarts]#  
 +
</pre>
 +
Now, assign the new kickstart to the profile we just created/copied:
-
[root@test-plccg kickstarts]# cobbler profile edit --name=ComputeNodes --kickstart=/var/lib/cobbler/kickstarts/Compute.ks
+
 
-
[root@test-plccg kickstarts]#
+
<pre>
 +
[root@headnode kickstarts]# cobbler profile edit --name=ComputeNodes --kickstart=/var/lib/cobbler/kickstarts/Compute.ks
 +
</pre>
Verify the change:-
Verify the change:-
-
 
+
<pre>
-
[root@test-plccg kickstarts]# cobbler profile report --name=ComputeNodes | grep kickstart
+
[root@headnode kickstarts]# cobbler profile report --name=ComputeNodes | grep kickstart
kickstart            : /var/lib/cobbler/kickstarts/Compute.ks
kickstart            : /var/lib/cobbler/kickstarts/Compute.ks
-
[root@test-plccg kickstarts]#   
+
[root@headnode kickstarts]#   
 +
</pre>
 +
= Define Systems =
 +
Now define/ add compute nodes. Remember, that my compute nodes have an IPMI capable interface, which have an out-of-band IP (separate network IP), than the access / production network. Right now I am only adding systems with the most basic information:
-
Now define/ add compute nodes:-
+
<pre>
 +
[root@headnode ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6A:27:91
 +
[root@headnode ~]# cobbler system add --name=compute2 --profile=ComputeNodes --mac=00:1D:09:6B:06:81
 +
</pre>
-
[root@test-plccg ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6A:27:91
 
-
[root@test-plccg ~]# cobbler system add --name=compute2 --profile=ComputeNodes --mac=00:1D:09:6B:06:81
+
<b>Note:</b> In case you try to make a duplicate entry:-
-
[root@test-plccg ~]#
+
-
 
+
<pre>
-
------------- Note ----------
+
[root@headnode ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6B:06:7F
-
In case you try to make a duplicate entry:-
+
-
 
+
-
[root@test-plccg ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6B:06:7F
+
An object already exists with that name.  Try 'edit'?
An object already exists with that name.  Try 'edit'?
 +
</pre>
-
------------------------------
 
-
[root@test-plccg ~]# cobbler system list
+
Check cobbler system list:
 +
<pre>
 +
[root@headnode ~]# cobbler system list
compute1
compute1
compute2
compute2
-
[root@test-plccg ~]#  
+
[root@headnode ~]#  
-
 
+
</pre>
-
You can check report of the nodes now. I am only interested in few parameters.
+
Looks good. You can check report of the nodes now. I am only interested in few parameters.
-
[root@test-plccg ~]# cobbler system report | egrep "system|profile|power type|interface|mac"
+
<pre>
 +
[root@headnode ~]# cobbler system report | egrep -i "system|profile|power type|interface|mac"
system                : compute1
system                : compute1
profile              : ComputeNodes
profile              : ComputeNodes
Line 432: Line 553:
interface        : eth0
interface        : eth0
   mac address    : 00:1D:09:6B:06:7F
   mac address    : 00:1D:09:6B:06:7F
-
[root@test-plccg ~]#                       
+
[root@headnode ~]#                       
 +
</pre>
-
I need to change the interface from eth0 to eth1.  
+
I need to change the interface from eth0 to eth1, because my nodes are connected to  access network through eth1. Whereas eth1 is the management / IPMI / BMC network.
 +
I can simply delete these two systems and recreate them without a hassle:
 +
<pre>
 +
[root@headnode ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6A:27:91 --hostname=compute1.example.com --interface=eth1  --static=1 --ip=192.168.1.1 --subnet=255.255.255.0 --kopts=noipv6  --power-address=192.168.0.1 --power-user=root --power-pass=ipmisecret
-
[root@test-plccg ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6A:27:91 --hostname=compute1.example.com --interface=eth1 --ip=192.168.1.1 --subnet=255.255.255.0 --kopts=noipv6 --power-address=192.168.0.1 --power-user=root --power-pass=power
+
[root@headnode ~]# cobbler system add --name=compute2 --profile=ComputeNodes --mac=00:1D:09:6B:06:81 --hostname=compute2.example.com --interface=eth1 --static=1 --ip=192.168.1.2 --subnet=255.255.255.0 --kopts=noipv6 --power-address=192.168.0.2  --power-user=root --power-pass=ipmisecret
 +
[root@headnode ~]#
-
[root@test-plccg ~]# cobbler system add --name=compute2 --profile=ComputeNodes --mac=00:1D:09:6B:06:81 --hostname=compute2.example.com --interface=eth1 --ip=192.168.1.2 --subnet=255.255.255.0 --kopts=noipv6 --power-address=192.168.0.2  --power-user=root --power-pass=power
+
Note: The BMC interface has the user name "root" and password "ipmisecret", are defined in each node's BMC configuration in BIOS.
-
[root@test-plccg ~]#
+
</pre>
 +
Now once you modify anything in the cobbler system, you will need to do a "cobbler sync", which will make necessary changes in kickstart and tftp files and restart dhcpd. (I wasted one full day browsing the net, to figure out this one!)
 +
 +
<pre>
 +
[root@headnode ~]# cobbler sync
 +
Shutting down dhcpd:                                      [  OK  ]
 +
Starting dhcpd:                                            [  OK  ]
 +
[root@headnode ~]#
 +
</pre>
 +
 +
And if you make any changes in cobbler's /etc/cobbler/settings file or any of it's subsystem, you will need to restart cobblerd service.
 +
 +
<pre>
 +
service cobblerd restart
 +
cobbler sync
 +
</pre>
-
[root@test-plccg ~]# cobbler system report | egrep "system|profile|power type|interface|mac"
 
-
system                : compute1
 
-
profile              : ComputeNodes
 
-
power type            : ipmitool
 
-
interface        : eth0
 
-
  mac address    : 00:1D:09:6A:27:8F
 
-
system                : compute2
 
-
profile              : ComputeNodes
 
-
power type            : ipmitool
 
-
interface        : eth0
 
-
  mac address    : 00:1D:09:6B:06:7F
 
-
[root@test-plccg ~]#
 
-
[root@test-plccg ~]# cat /tftpboot/pxelinux.cfg/01-00-1d-09-6a-27-8f
+
If you look at the TFTP file created by cobbler for compute1, it looks like this:
 +
<pre>
 +
[root@headnode ~]# cat /tftpboot/pxelinux.cfg/01-00-1d-09-6a-27-8f
default linux
default linux
prompt 0
prompt 0
Line 467: Line 598:
         append initrd=/images/Centos5.4-x86_64/initrd.img ksdevice=bootif lang=  kssendmac text  ks=http://192.168.1.200/cblr/svc/op/ks/system/compute1
         append initrd=/images/Centos5.4-x86_64/initrd.img ksdevice=bootif lang=  kssendmac text  ks=http://192.168.1.200/cblr/svc/op/ks/system/compute1
-
[root@test-plccg ~]#
+
[root@headnode ~]#
 +
<pre>
 +
The resultant dhcpd.conf file looks like this:
-
service cobblerd restart
+
<pre>
-
 
+
cat /etc/dhcpd.conf
-
[root@test-plccg ~]# service dhcpd restart
+
-
Starting dhcpd:                                            [FAILED]
+
-
[root@test-plccg ~]# 
+
-
 
+
-
 
+
-
Wasted all my day. This is the fix:-
+
-
 
+
-
[root@test-plccg ~]# cobbler sync
+
-
Shutting down dhcpd:                                      [  OK  ]
+
-
Starting dhcpd:                                            [  OK  ]
+
-
 
+
-
 
+
-
[root@test-plccg ~]# cat /etc/dhcpd.conf
+
# ******************************************************************
# ******************************************************************
# Cobbler managed dhcpd.conf file
# Cobbler managed dhcpd.conf file
Line 521: Line 641:
}
}
-
[root@test-plccg ~]#   
+
[root@headnode ~]#   
 +
</pre>
-
[root@test-plccg ~]# ls  /tftpboot/pxelinux.cfg/ -l
+
For two nodes, I had the following TFTP files:
 +
<pre>
 +
[root@headnode ~]# ls  /tftpboot/pxelinux.cfg/ -l
total 12
total 12
-rw-r--r-- 1 root root  272 Feb  8 13:18 01-00-1d-09-6a-27-91
-rw-r--r-- 1 root root  272 Feb  8 13:18 01-00-1d-09-6a-27-91
-rw-r--r-- 1 root root  272 Feb  8 13:18 01-00-1d-09-6b-06-81
-rw-r--r-- 1 root root  272 Feb  8 13:18 01-00-1d-09-6b-06-81
-rw-r--r-- 1 root root 1091 Feb  8 13:18 default
-rw-r--r-- 1 root root 1091 Feb  8 13:18 default
-
[root@test-plccg ~]#
+
[root@headnode ~]#
 +
</pre>
-
A bit of power issue.
+
If your systems have IPMI capable interfaces, then you must make sure that you have ipmitools, ipmiconsole and ipmipower installed.
-
ipmitool, which is supported by CentOS, could not be found on the system. Instead, ipmipower was there, from freeipmi-rpm.
+
<pre>
 +
[root@headnode ~]# yum -y install OpenIPMI-tools freeipmi-*
-
/etc/cobbler/settings mention something about the power scripts/templates.  
+
[root@headnode ~]# rpm -ql OpenIPMI-tools
-
These are in /etc/cobbler/power directory.
+
/etc/rc.d/init.d/ipmievd
 +
/etc/sysconfig/ipmievd
 +
/usr/bin/ipmitool
 +
</pre>
-
[root@test-plccg ~]# cd /etc/cobbler/power/
 
-
[root@test-plccg power]# ls
 
-
power_apc_snmp.template    power_ether_wake.template  power_ipmitool.template  power_wti.template
 
-
power_bladecenter.template  power_ilo.template        power_lpar.template
 
-
power_bullpap.template      power_integrity.template  power_rsa.template
 
-
power_drac.template        power_ipmilan.template    power_virsh.template
 
 +
Time to reboot these nodes and see if they are able to install. However, make sure that their netboot switch is enabled.
-
[root@test-plccg power]# cat power_ipmitool.template
+
<pre>
-
/usr/bin/ipmitool -H $power_address -U $power_user -P $power_pass power $power_mode
+
[root@headnode power]# cobbler system edit --name=compute1 --netboot=1
-
[root@test-plccg power]#
+
[root@headnode power]# cobbler system edit --name=compute2 --netboot=1
-
 
+
-
The syntax of ipmitool and ipmipower are a bit different so this is not do-able. (Probably).
+
-
 
+
-
 
+
-
 
+
-
[root@test-plccg ~]# yum -y install OpenIPMI-tools
+
-
 
+
-
 
+
-
[root@test-plccg ~]# rpm -ql OpenIPMI-tools
+
-
/etc/rc.d/init.d/ipmievd
+
-
/etc/sysconfig/ipmievd
+
-
/usr/bin/ipmitool
+
-
 
+
-
Ahaa!!!
+
-
Time to reboot these nodes and see if they are installed:
 
-
[root@test-plccg power]# cobbler report --what=systems | egrep -i "hostname|netboot"
+
[root@headnode power]# cobbler report --what=systems | egrep -i "hostname|netboot"
hostname              : compute1.example.com
hostname              : compute1.example.com
netboot enabled?      : True
netboot enabled?      : True
hostname              : compute2.example.com
hostname              : compute2.example.com
netboot enabled?      : True
netboot enabled?      : True
-
[root@test-plccg power]#  
+
[root@headnode power]#  
 +
</pre>
 +
There is no "install" command. Instead all the installation mechanism has been made ready. You need to reboot the nodes to get installed.
-
[root@test-plccg power]# cobbler system reboot --name=compute1
+
<pre>
 +
[root@headnode power]# cobbler system reboot --name=compute1
cobbler power configuration is:
cobbler power configuration is:
Line 584: Line 695:
       id    :
       id    :
-
- /usr/bin/ipmitool -H 192.168.0.1 -U root -P power power off
+
- /usr/bin/ipmitool -H 192.168.0.1 -U root -P ipmisecret power off
Chassis Power Control: Down/Off
Chassis Power Control: Down/Off
cobbler power configuration is:
cobbler power configuration is:
Line 593: Line 704:
       id    :
       id    :
-
- /usr/bin/ipmitool -H 192.168.0.1 -U root -P power power on
+
- /usr/bin/ipmitool -H 192.168.0.1 -U root -P ipmisecret power on
Chassis Power Control: Up/On
Chassis Power Control: Up/On
-
[root@test-plccg power]#     
+
[root@headnode power]#     
 +
</pre>
-
After reboot of node1, the installation started. However it got stuck in an install loop.
 
-
This happened, because PXE is the first boot device in the nodes.
 
-
[root@test-plccg power]# cobbler report --what=systems | egrep -i "hostname|netboot"
+
If, after reboot of node, the installation starts and finishes and starts again, it means your nodes are stuck in an install loop. This happens, because PXE is the first boot device in the nodes, which is the required boot sequence in HPCC scenarios. What you need to do is to make sure that a particular setting is enabled in the /etc/cobblet/settings file. Without setting that, the netboot flag will remain switched on in cobbler configuration and the nodes will keep rebooting and keep installing.
 +
 
 +
<pre>
 +
[root@headnode power]# cobbler report --what=systems | egrep -i "hostname|netboot"
hostname              : compute1.example.com
hostname              : compute1.example.com
netboot enabled?      : True
netboot enabled?      : True
hostname              : compute2.example.com
hostname              : compute2.example.com
netboot enabled?      : True
netboot enabled?      : True
-
[root@test-plccg power]#   
+
[root@headnode power]#   
 +
</pre>
For such situations, there is a flag in /etc/cobbler/settings, which turns off the netboot flag, when the installation is done. It is "pxe_just_once" and is set to 0 by default. It needs to be turned to 1 in this situation.
For such situations, there is a flag in /etc/cobbler/settings, which turns off the netboot flag, when the installation is done. It is "pxe_just_once" and is set to 0 by default. It needs to be turned to 1 in this situation.
 +
<pre>
pxe_just_once: 1
pxe_just_once: 1
 +
</pre>
-
This worked!
+
= Day 3. Few other issues.=
-
===========================================
+
My nodes installed properly and booted properly. However they were taking too much time in trying to obtain IP for eth0 interface through dhcp. Since there is no dhcp on eth0 and I don't need to enable the eth0 interface in OS, because it is enabled and properly configured in the BIOS already, I had to find a way to fix that. Here are four items which I wanted to know the solution:
-
Now the few other issues.  
+
1) Static IP for eth1. Means, after installation of a node is complete, it should have a static IP configured in it instead of asking that from DHCP server.
 +
2) Disable eth0, so it does not wait for dhcp when the node reboots from local OS/disk.  
 +
3) Console redirection during TFTP and installation. I want to see the console of the node on my headnode, instead of manually seeing it on the KVM. (It does not make sense to run to the data center everytime you need to look at screen of a node.)
 +
4) How to check if the nodes are installed? Pretty important!
-
1) Static IP for eth1, and see if it still installs from DHCP server.
 
-
2) disable eth0 so it does not wait for dhcp when the node reboots in to local os /disk.
 
-
3) How to check if the nodes are installed?
 
-
4) Console redirection during TFTP and installation.
 
-
 
-
-------------------
 
1) Static IP:-
1) Static IP:-
-
[root@test-plccg ~]# cobbler system edit --name compute1 --interface=eth1 --static=1
+
Simply add a --static=1 to either the cobbler system add command, or modify an already defined system with this setting.
 +
<pre>
 +
[root@headnode ~]# cobbler system edit --name=compute1 --interface=eth1 --static=1
 +
</pre>
 +
 
This is the solution for setting static IP, post-reboot. The installation got done through PXE (good).
This is the solution for setting static IP, post-reboot. The installation got done through PXE (good).
-
Here is what cobbler did:-
+
Here is what cobbler did, after the node is installed and rebooted from local disk:-
 +
<pre>
[root@compute1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
[root@compute1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
DEVICE=eth1
Line 640: Line 758:
NETMASK=255.255.255.0
NETMASK=255.255.255.0
[root@compute1 ~]#
[root@compute1 ~]#
 +
</pre>
 +
This is exactly what I wanted. Good.
-
2) disable eth0
 
-
[root@test-plccg ~]# cobbler system edit --name=compute1 --delete-interface=eth0
 
-
(I don't know, if it will work)
 
-
This did not work. Lets set eth0 as static. so it won't waste time on waiting for DHCP and failing.
+
2) Disable eth0. For this, I used the same technique. I just mentioned that it is also static, and did not provide any IP for it. That is because the (out of band) IP is already configured in the node's BIOS.
-
[root@test-plccg ~]# cobbler system edit --name compute1 --interface=eth0 --static=1
+
<pre>
 +
[root@headnode ~]# cobbler system edit --name=compute1 --interface=eth0 --static=1
-
[root@test-plccg ~]# cobbler report system --name=compute1
+
[root@headnode ~]# cobbler report system --name=compute1
system                : compute1
system                : compute1
profile              : ComputeNodes
profile              : ComputeNodes
Line 670: Line 788:
   dhcp tag      :
   dhcp tag      :
-
[root@test-plccg ~]#                           
+
[root@headnode ~]#                           
 +
</pre>
 +
 
 +
 
 +
Note that --delete-interface did NOT do the trick.
 +
 
 +
<pre>
 +
[root@headnode ~]# cobbler system edit --name=compute1 --delete-interface=eth0
 +
 
 +
This did not work.
 +
</pre>
 +
 
-
This time it worked. I have lovely results!
+
This time it worked. I have lovely results after the nodes are installed and rebooted!
 +
<pre>
[root@compute1 ~]# ifconfig
[root@compute1 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1D:09:6A:27:8F
eth0      Link encap:Ethernet  HWaddr 00:1D:09:6A:27:8F
Line 700: Line 830:
NETMASK=255.255.255.0
NETMASK=255.255.255.0
[root@compute1 ~]#  
[root@compute1 ~]#  
-
 
+
</pre>
Notice that Cobbler set BOOTPROTO=none for eth0.  
Notice that Cobbler set BOOTPROTO=none for eth0.  
-
However the node was not able to ping any other node on IPMI interface. hmmmmmmm.
 
 +
However the node WILL NOT ping any other node on IPMI interface. This is only controllable/ do-able in one direction, from headnode towards compute node. Remember that the headnode's management interface HAS an out of band IP configured in the OS.
 +
 +
Thus, don't be alarmed if you see the following, while you are logged in to any of the compute nodes:
 +
<pre>
[root@compute1 ~]# ping 192.168.0.200
[root@compute1 ~]# ping 192.168.0.200
connect: Network is unreachable
connect: Network is unreachable
Line 712: Line 845:
connect: Network is unreachable
connect: Network is unreachable
[root@compute1 ~]#
[root@compute1 ~]#
 +
</pre>
-
-------------------------
+
End of day 3.
-
Trying this:-
+
= Day 4. Console redirection. =
-
[root@test-plccg ~]# cobbler system edit --name=compute1 --delete-interface=eth0
+
-
This did not help prevent eth0 asking for dhcp / IP.
+
For (3) We set certain parameters in the distro and profile options.
-
 
+
<pre>
-
----------------------
+
[root@headnode ~]# cobbler distro list
-
 
+
-
Day n+1 (3?)
+
-
--------------
+
-
 
+
-
How to pass console redirection options to TFTP file:-
+
-
 
+
-
[root@test-plccg ~]# cobbler distro list
+
Centos5.4-x86_64
Centos5.4-x86_64
Centos5.4-xen-x86_64
Centos5.4-xen-x86_64
-
[root@test-plccg ~]#
+
[root@headnode ~]#
-
 
+
</pre>
Centos5.4-x86_64 is the distro we are using for provisioning the nodes.
Centos5.4-x86_64 is the distro we are using for provisioning the nodes.
-
[root@test-plccg ~]# cobbler distro report --name=Centos5.4-x86_64
+
<pre>
 +
[root@headnode ~]# cobbler distro report --name=Centos5.4-x86_64
distro              : Centos5.4-x86_64
distro              : Centos5.4-x86_64
architecture        : x86_64
architecture        : x86_64
Line 755: Line 882:
redhat mgmt server  : <<inherit>>
redhat mgmt server  : <<inherit>>
template files      : {}
template files      : {}
 +
[root@headnode ~]#
-
[root@test-plccg ~]#
 
-
[root@test-plccg ~]# cobbler distro edit --name=Centos5.4-x86_64 --kopts="console=ttyS1,115200"
+
[root@headnode ~]# cobbler distro edit --name=Centos5.4-x86_64 --kopts="console=ttyS1,115200"
-
[root@test-plccg ~]# cobbler distro report --name=Centos5.4-x86_64
+
 
 +
[root@headnode ~]# cobbler distro report --name=Centos5.4-x86_64
distro              : Centos5.4-x86_64
distro              : Centos5.4-x86_64
architecture        : x86_64
architecture        : x86_64
Line 780: Line 908:
template files      : {}
template files      : {}
-
[root@test-plccg ~]#
+
[root@headnode ~]#
-
 
+
</pre>
Now check the profiles:-
Now check the profiles:-
-
[root@test-plccg ~]# cobbler profile list
+
<pre>
 +
[root@headnode ~]# cobbler profile list
Centos5.4-x86_64
Centos5.4-x86_64
Centos5.4-xen-x86_64
Centos5.4-xen-x86_64
ComputeNodes
ComputeNodes
rescue-Centos5.4-x86_64
rescue-Centos5.4-x86_64
-
[root@test-plccg ~]#
+
[root@headnode ~]#
 +
</pre>
We are interested in profile named "ComputeNodes":-
We are interested in profile named "ComputeNodes":-
-
[root@test-plccg ~]# cobbler profile report --name=ComputeNodes
+
<pre>
 +
[root@headnode ~]# cobbler profile report --name=ComputeNodes
profile              : ComputeNodes
profile              : ComputeNodes
distro              : Centos5.4-x86_64
distro              : Centos5.4-x86_64
Line 824: Line 955:
virt type            : qemu
virt type            : qemu
-
[root@test-plccg ~]#     
+
[root@headnode ~]#     
 +
[root@headnode ~]# cobbler profile edit --name=ComputeNodes --ksmeta="console=ttyS1,115200"
-
[root@test-plccg ~]# cobbler profile edit --name=ComputeNodes --ksmeta="console=ttyS1,115200"
 
-
 
+
[root@headnode ~]# cobbler sync
-
[root@test-plccg ~]# cobbler sync
+
Starting dhcpd:                                            [  OK  ]
Starting dhcpd:                                            [  OK  ]
-
[root@test-plccg ~]#  
+
[root@headnode ~]#  
 +
</pre>
 +
 
 +
Try re-provisioning the node and see if it works.
-
[root@test-plccg ~]# cobbler system edit --name compute1 --netboot-enabled=1
+
<pre>
 +
[root@headnode ~]# cobbler system edit --name=compute1 --netboot-enabled=1
-
[root@test-plccg ~]# cobbler sync
+
[root@headnode ~]# cobbler sync
Shutting down dhcpd:                                      [  OK  ]
Shutting down dhcpd:                                      [  OK  ]
Starting dhcpd:                                            [  OK  ]
Starting dhcpd:                                            [  OK  ]
-
[root@test-plccg ~]# cat /tftpboot/pxelinux.cfg/01-00-1d-09-6a-27-91
+
[root@headnode ~]# cat /tftpboot/pxelinux.cfg/01-00-1d-09-6a-27-91
default linux
default linux
prompt 0
prompt 0
Line 850: Line 984:
         append initrd=/images/Centos5.4-x86_64/initrd.img ksdevice=bootif lang=  console=ttyS1,115200 text noipv6 kssendmac  ks=http://192.168.1.200/cblr/svc/op/ks/system/compute1
         append initrd=/images/Centos5.4-x86_64/initrd.img ksdevice=bootif lang=  console=ttyS1,115200 text noipv6 kssendmac  ks=http://192.168.1.200/cblr/svc/op/ks/system/compute1
-
[root@test-plccg ~]#
+
[root@headnode ~]#
 +
</pre>
-
--------------------
+
Notice the console option in the newly generated TFTP file above.
-
ipmiconsole
+
-
[root@test-plccg ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
+
= ipmiconsole =  
-
ipmiconsole_submit_block: internal system error
+
-
[root@test-plccg ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=redhat
+
Use the ipmiconsole command to open console of a remote node on the headnode.
 +
 
 +
<pre>
 +
[root@headnode ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
 +
[SOL established]
 +
</pre>
 +
 
 +
Note, if you get an error such as below, that means your headnode, does not have BMC interface correctly configured.  Assign it an IP from your BMC network and bring the interface up. It should work.
 +
 
 +
The error:
 +
<pre>
 +
[root@headnode ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
ipmiconsole_submit_block: internal system error
ipmiconsole_submit_block: internal system error
 +
</pre>
 +
-
[root@test-plccg ~]# ifconfig
+
The reason:
 +
<pre>
 +
[root@headnode ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BB
eth0      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BB
           inet6 addr: fe80::21d:9ff:fe69:f3bb/64 Scope:Link
           inet6 addr: fe80::21d:9ff:fe69:f3bb/64 Scope:Link
Line 872: Line 1,020:
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           inet6 addr: ::1/128 Scope:Host
 +
</pre>
-
[root@test-plccg ~]# ifconfig eth0 192.168.0.201 netmask 255.255.255.0 up
+
The fix:
-
[root@test-plccg ~]# ifconfig
+
<pre>
 +
[root@headnode ~]# ifconfig eth0 192.168.0.201 netmask 255.255.255.0 up
 +
 
 +
[root@headnode ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BB
eth0      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BB
           inet addr:192.168.0.201  Bcast:192.168.0.255  Mask:255.255.255.0
           inet addr:192.168.0.201  Bcast:192.168.0.255  Mask:255.255.255.0
Line 887: Line 1,039:
           inet6 addr: ::1/128 Scope:Host
           inet6 addr: ::1/128 Scope:Host
-
[root@test-plccg ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
+
[root@headnode ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
[SOL established]
[SOL established]
-
-----------------
+
</pre>
 +
 
 +
= power on/off =
 +
Use ipmipower command to perform power operations on the nodes.
 +
 
 +
<pre>
 +
[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --on
 +
192.168.0.1: ok
 +
</pre>
Also if the node is off, cycle may not work:-
Also if the node is off, cycle may not work:-
-
[root@test-plccg ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --cycle
+
<pre>
 +
[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --cycle
192.168.0.1: operation invalid
192.168.0.1: operation invalid
-
[root@test-plccg ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --off
+
[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --off
192.168.0.1: ok
192.168.0.1: ok
-
[root@test-plccg ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --on
+
[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --on
192.168.0.1: ok
192.168.0.1: ok
-
[root@test-plccg ~]#
+
[root@headnode ~]#
 +
</pre>
-
-----------------------------------
 
 +
= Problem with console entries =
-
When using console=ttyS1,57600n8r , we get a node install time slowed down 5 times. Without it, the node install time was 1 minute. With it the node install time reached 5+ minutes.
+
When using console=ttyS1,57600n8r , we get a node install time slowed down 5 times. Without it, the node install time was 1 minute. With it, the node install time reached 5+ minutes.
Using 115200 as console bnaud rate shows too much garbage on ipmiconsole as soon as the TFTP boot image is read. Before that, the BIOS display part is ok.
Using 115200 as console bnaud rate shows too much garbage on ipmiconsole as soon as the TFTP boot image is read. Before that, the BIOS display part is ok.
-
Why is that?
 
-
 
-
--------------
 
-
Why two console= options in Scali and CVOS?
 
-
 
-
Gilbert suggests to use console ttyS0 as well, along ttyS1. This is because ttyS1 is redirected to SOL. Whereas ttyS0 is not redirected, thus the messages should appear on the normal VGA console as well.
 
-
 
-
This did not work. I lost console on both the IPMI console and the node itself.
 
-
 
-
----------------
 
How to get rid of garbage on console when using 115200 baud rate?
How to get rid of garbage on console when using 115200 baud rate?
-
 
-
==========================================================
 
-
 
-
Day 2:
 
-
 
-
Cobbler howto-2
 
-
=============
 
-
 
-
 
-
[root@test-plccg ~]# cobbler check
 
-
The following potential problems were detected:
 
-
#0: service dhcpd is not running
 
-
#1: since iptables may be running, ensure 69, 80, and 25151 are unblocked
 
-
#2: yum-utils need to be at least version 1.1.17 for reposync -l, current version is 1.1.16
 
-
#3: The default password used by the sample templates for newly installed machines (default_password_crypted in /etc/cobbler/settings) is still set to 'cobbler' and should be changed
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
 
-
[root@test-plccg ~]# cobbler list
 
-
distro Centos5.4-x86_64
 
-
  profile Centos5.4-x86_64
 
-
  profile rescue-Centos5.4-x86_64
 
-
distro Centos5.4-xen-x86_64
 
-
  profile Centos5.4-xen-x86_64
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
[root@test-plccg ~]# cobbler distro list
 
-
Centos5.4-x86_64
 
-
Centos5.4-xen-x86_64
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
[root@test-plccg ~]# cobbler profile list
 
-
Centos5.4-x86_64
 
-
Centos5.4-xen-x86_64
 
-
rescue-Centos5.4-x86_64
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
No systems are registered with cobbler at the moment:-
 
-
 
-
[root@test-plccg ~]# cobbler system list
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
Lets create a profile for compute-nodes:-
 
-
 
-
[root@test-plccg ~]# cobbler profile copy --name=Centos5.4-x86_64 --newname=ComputeNodes
 
-
 
-
 
-
[root@test-plccg ~]# cobbler profile list
 
-
Centos5.4-x86_64
 
-
Centos5.4-xen-x86_64
 
-
ComputeNodes                # <------------ Newly added profile!
 
-
rescue-Centos5.4-x86_64
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
[root@test-plccg ~]# cobbler profile report --name=ComputeNodes
 
-
profile              : ComputeNodes
 
-
distro              : Centos5.4-x86_64
 
-
comment              :
 
-
created              : Tue Feb  2 09:06:50 2010
 
-
dhcp tag            : default
 
-
enable menu          : True
 
-
kernel options      : {}
 
-
kickstart            : /var/lib/cobbler/kickstarts/default.ks  # < ------------ Notice this default Kickstart.
 
-
ks metadata          : {}
 
-
mgmt classes        : []
 
-
modified            : Tue Feb  2 09:06:50 2010
 
-
name servers        : []
 
-
name servers search  : []
 
-
owners              : ['admin']
 
-
post kernel options  : {}
 
-
redhat mgmt key      : <<inherit>>
 
-
redhat mgmt server  : <<inherit>>
 
-
repos                : []
 
-
server              : <<inherit>>
 
-
template_files      : {}
 
-
virt bridge          : xenbr0
 
-
virt cpus            : 1
 
-
virt file size      : 5
 
-
virt path            :
 
-
virt ram            : 512
 
-
virt type            : qemu
 
-
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
[root@test-plccg ~]# cat /var/lib/cobbler/kickstarts/default.ks
 
-
# this file intentionally left blank
 
-
# admins:  edit it as you like, or leave it blank for non-interactive install
 
-
 
-
 
-
[root@test-plccg ~]# cat /var/lib/cobbler/kickstarts/sample.ks
 
-
#platform=x86, AMD64, or Intel EM64T
 
-
# System authorization information
 
-
auth  --useshadow  --enablemd5
 
-
# System bootloader configuration
 
-
bootloader --location=mbr
 
-
# Partition clearing information
 
-
clearpart --all --initlabel
 
-
# Use text mode install
 
-
text
 
-
# Firewall configuration
 
-
firewall --enabled
 
-
# Run the Setup Agent on first boot
 
-
firstboot --disable
 
-
# System keyboard
 
-
keyboard us
 
-
# System language
 
-
lang en_US
 
-
# Use network installation
 
-
url --url=$tree
 
-
# If any cobbler repo definitions were referenced in the kickstart profile, include them here.
 
-
$yum_repo_stanza
 
-
# Network information
 
-
$SNIPPET('network_config')
 
-
# Reboot after installation
 
-
reboot
 
-
 
-
#Root password
 
-
rootpw --iscrypted $default_password_crypted
 
-
# SELinux configuration
 
-
selinux --disabled
 
-
# Do not configure the X Window System
 
-
skipx
 
-
# System timezone
 
-
timezone  America/New_York
 
-
# Install OS instead of upgrade
 
-
install
 
-
# Clear the Master Boot Record
 
-
zerombr
 
-
# Allow anaconda to partition the system as needed
 
-
autopart
 
-
 
-
 
-
%pre
 
-
$SNIPPET('log_ks_pre')
 
-
$kickstart_start
 
-
$SNIPPET('pre_install_network_config')
 
-
# Enable installation monitoring
 
-
$SNIPPET('pre_anamon')
 
-
 
-
%packages
 
-
$SNIPPET('func_install_if_enabled')
 
-
 
-
%post
 
-
$SNIPPET('log_ks_post')
 
-
# Start yum configuration
 
-
$yum_config_stanza
 
-
# End yum configuration
 
-
$SNIPPET('post_install_kernel_options')
 
-
$SNIPPET('post_install_network_config')
 
-
$SNIPPET('func_register_if_enabled')
 
-
$SNIPPET('download_config_files')
 
-
$SNIPPET('koan_environment')
 
-
$SNIPPET('redhat_register')
 
-
$SNIPPET('cobbler_register')
 
-
# Enable post-install boot notification
 
-
$SNIPPET('post_anamon')
 
-
# Start final steps
 
-
$kickstart_done
 
-
# End final steps
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
 
-
[root@test-plccg ~]# cd /var/lib/cobbler/kickstarts/
 
-
 
-
[root@test-plccg kickstarts]# cp sample.ks Compute.ks
 
-
 
-
[root@test-plccg kickstarts]# vi Compute.ks
 
-
#platform=x86, AMD64, or Intel EM64T
 
-
# System authorization information
 
-
auth  --useshadow  --enablemd5
 
-
# System bootloader configuration
 
-
bootloader --location=mbr
 
-
# Partition clearing information
 
-
clearpart --all --initlabel
 
-
# Use text mode install
 
-
text
 
-
# Firewall configuration
 
-
firewall --disabled
 
-
# Run the Setup Agent on first boot
 
-
firstboot --disable
 
-
# System keyboard
 
-
keyboard us
 
-
# System language
 
-
lang en_US
 
-
# Use network installation
 
-
url --url=$tree
 
-
# If any cobbler repo definitions were referenced in the kickstart profile, include them here.
 
-
$yum_repo_stanza
 
-
# Network information
 
-
$SNIPPET('network_config')
 
-
# Reboot after installation
 
-
reboot
 
-
 
-
#Root password
 
-
rootpw --iscrypted $default_password_crypted
 
-
# SELinux configuration
 
-
selinux --disabled
 
-
# Do not configure the X Window System
 
-
skipx
 
-
# System timezone
 
-
timezone  Asia/Riyadh
 
-
# Install OS instead of upgrade
 
-
install
 
-
# Clear the Master Boot Record
 
-
zerombr
 
-
# Allow anaconda to partition the system as needed
 
-
autopart
 
-
 
-
 
-
%pre
 
-
$SNIPPET('log_ks_pre')
 
-
$kickstart_start
 
-
$SNIPPET('pre_install_network_config')
 
-
# Enable installation monitoring
 
-
$SNIPPET('pre_anamon')
 
-
 
-
%packages
 
-
@base
 
-
$SNIPPET('func_install_if_enabled')
 
-
 
-
%post
 
-
$SNIPPET('log_ks_post')
 
-
# Start yum configuration
 
-
$yum_config_stanza
 
-
# End yum configuration
 
-
$SNIPPET('post_install_kernel_options')
 
-
$SNIPPET('post_install_network_config')
 
-
$SNIPPET('func_register_if_enabled')
 
-
$SNIPPET('download_config_files')
 
-
$SNIPPET('koan_environment')
 
-
$SNIPPET('redhat_register')
 
-
$SNIPPET('cobbler_register')
 
-
# Enable post-install boot notification
 
-
$SNIPPET('post_anamon')
 
-
# Start final steps
 
-
$kickstart_done
 
-
# End final steps
 
-
[root@test-plccg kickstarts]#
 
-
 
-
 
-
[root@test-plccg kickstarts]# cobbler profile edit --name=ComputeNodes --kickstart=/var/lib/cobbler/kickstarts/Compute.ks
 
-
[root@test-plccg kickstarts]#
 
-
 
-
 
-
Verify the change:-
 
-
 
-
[root@test-plccg kickstarts]# cobbler profile report --name=ComputeNodes | grep kickstart
 
-
kickstart            : /var/lib/cobbler/kickstarts/Compute.ks
 
-
[root@test-plccg kickstarts]# 
 
-
 
-
 
-
Now define/ add compute nodes:-
 
-
 
-
[root@test-plccg ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6A:27:8F
 
-
 
-
[root@test-plccg ~]# cobbler system add --name=compute2 --profile=ComputeNodes --mac=00:1D:09:6B:06:7F
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
------------- Note ----------
 
-
In case you try to make a duplicate entry:-
 
-
 
-
[root@test-plccg ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6B:06:7F
 
-
An object already exists with that name.  Try 'edit'?
 
-
 
-
------------------------------
 
-
 
-
[root@test-plccg ~]# cobbler system list
 
-
compute1
 
-
compute2
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
You can check report of the nodes now. I am only interested in few parameters.
 
-
 
-
[root@test-plccg ~]# cobbler system report | egrep "system|profile|power type|interface|mac"
 
-
system                : compute1
 
-
profile              : ComputeNodes
 
-
power type            : ipmitool
 
-
interface        : eth0
 
-
  mac address    : 00:1D:09:6A:27:8F
 
-
system                : compute2
 
-
profile              : ComputeNodes
 
-
power type            : ipmitool
 
-
interface        : eth0
 
-
  mac address    : 00:1D:09:6B:06:7F
 
-
[root@test-plccg ~]#                     
 
-
 
-
I need to change the interface from eth0 to eth1.
 
-
 
-
[root@test-plccg ~]# cobbler system edit --name=compute1 --interface=eth1  --power-type=ipmitool --power-user=root --power-pass=calvin
 
-
 
-
[root@test-plccg ~]# cobbler system edit --name=compute2 --interface=eth1  --power-type=ipmitool --power-user=root --power-pass=calvin
 
-
[root@test-plccg ~]# 
 
-
 
-
 
-
[root@test-plccg ~]# cobbler system report | egrep "system|profile|power type|interface|mac"
 
-
system                : compute1
 
-
profile              : ComputeNodes
 
-
power type            : ipmitool
 
-
interface        : eth0
 
-
  mac address    : 00:1D:09:6A:27:8F
 
-
system                : compute2
 
-
profile              : ComputeNodes
 
-
power type            : ipmitool
 
-
interface        : eth0
 
-
  mac address    : 00:1D:09:6B:06:7F
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
[root@test-plccg ~]# cat /tftpboot/pxelinux.cfg/01-00-1d-09-6a-27-8f
 
-
default linux
 
-
prompt 0
 
-
timeout 1
 
-
label linux
 
-
        kernel /images/Centos5.4-x86_64/vmlinuz
 
-
        ipappend 2
 
-
        append initrd=/images/Centos5.4-x86_64/initrd.img ksdevice=bootif lang=  kssendmac text  ks=http://192.168.1.200/cblr/svc/op/ks/system/compute1
 
-
 
-
[root@test-plccg ~]#
 
-
 
-
 
-
service cobblerd restart
 
-
 
-
[root@test-plccg ~]# service dhcpd restart
 
-
Starting dhcpd:                                            [FAILED]
 
-
[root@test-plccg ~]# 
 
-
 
-
 
-
Wasted all my day. This is the fix:-
 
-
 
-
[root@test-plccg ~]# cobbler sync
 
-
Shutting down dhcpd:                                      [  OK  ]
 
-
Starting dhcpd:                                            [  OK  ]
 
-
 
-
 
-
[root@test-plccg ~]# cat /etc/dhcpd.conf
 
-
# ******************************************************************
 
-
# Cobbler managed dhcpd.conf file
 
-
# generated from cobbler dhcp.conf template (Tue Feb  2 11:47:35 2010)
 
-
# Do NOT make changes to /etc/dhcpd.conf. Instead, make your changes
 
-
# in /etc/cobbler/dhcp.template, as /etc/dhcpd.conf will be
 
-
# overwritten.
 
-
# ******************************************************************
 
-
 
-
ddns-update-style interim;
 
-
 
-
allow booting;
 
-
allow bootp;
 
-
 
-
ignore client-updates;
 
-
set vendorclass = option vendor-class-identifier;
 
-
 
-
subnet 192.168.1.0 netmask 255.255.255.0 {
 
-
    option routers            192.168.1.200;
 
-
    option domain-name-servers 192.168.1.200;
 
-
    option subnet-mask        255.255.255.0;
 
-
    range dynamic-bootp        192.168.1.1 192.168.1.10;
 
-
    filename                  "/pxelinux.0";
 
-
    default-lease-time        21600;
 
-
    max-lease-time            43200;
 
-
    next-server                192.168.1.200;
 
-
}
 
-
 
-
# group for Cobbler DHCP tag: default
 
-
group {
 
-
    host generic1 {
 
-
        hardware ethernet 00:1D:09:6B:06:91;
 
-
        filename "/pxelinux.0";
 
-
        next-server 192.168.1.200;
 
-
    }
 
-
}
 
-
 
-
[root@test-plccg ~]#
 

Current revision as of 12:15, 27 February 2010

Contents

Cobbler Howto

Cobbler install on Dell 1950 Servers

This setup consists of four Dell PowerEdge 1950 servers, connected to each other in a cluster form. The intended purpose of this excercise is to find a way to provision compute nodes, without hassle. This should be ideal for data centers (both HPCC and non-HPCC).


Help from:-


The system has Centos 5.4 x86_64 installed as base OS. You might want to update it before continuing. 

yum -y  update


After this you should make sure that you have the following installed, from your distribution media / repository.

  • httpd
  • tftp-server
  • xinetd
  • reposync (yum-utils provides reposync) 
yum -y install httpd yum-utils tftp-server xinetd dhcp

Make sure you are not running kernel-xen on your base OS. (Although it does not matter, but I just want to keep things simple). My scenario requires me not to run kernel-xen. (We are not using virtualization features. These are pure compute nodes.)

Using the link: https://fedorahosted.org/cobbler/wiki/DownloadInstructions

Setup Cobbler Repositories:- (Needs internet connection). 


[root@headnode ~]# vi /etc/yum.repos.d/CentOS-Base.repo 
. . .
. . . 
[cobbler-stable]
name=Cobbler-Stable
baseurl=http://download.fedora.redhat.com/pub/epel/5/x86_64

[cobbler-testing]
name=Cobbler-Testing
baseurl=http://download.fedora.redhat.com/pub/epel/testing/5/x86_64
. . .
. . .  
[root@headnode ~]#


Now, install cobbler:-

[root@headnode ~]# yum -y install cobbler koan cman


I have various CD images in my /data/cdimages location. I expanded the ISO image and copied it in CentOS-5.4-x86_64 . 

[root@headnode cdimages]# ls
CentOS-5.3-i386.DVD.iso        
CentOS-5.4-x86_64              
CentOS-5.4-x86_64-bin-DVD.iso  
[root@headnode cdimages]#

Assuming you have entire CD media copied in your /data/cdimages/CentOS-5.4-x86_64 , you are now ready to import the repository. 

cobbler import --mirror=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64

or 

cobbler import --path=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64

During import you should get some screen out put as: 

[root@headnode cdimages]# cobbler import --mirror=/data/cdimages/CentOS-5.4-x86_64 --name=Centos5.4-x86_64
. . .
 . . .

repodata/primary.sqlite.bz2                                                                                           
     2141520 100%    3.07MB/s    0:00:00 (xfer#3468, to-check=2/3478)                                                 
repodata/primary.xml.gz                                                                                               
     1189512 100%    1.67MB/s    0:00:00 (xfer#3469, to-check=1/3478)                                                 
repodata/repomd.xml                                                                                                   
        2142 100%    3.09kB/s    0:00:00 (xfer#3470, to-check=0/3478)                                                 

sent 4333936317 bytes  received 76408 bytes  39580024.89 bytes/sec
total size is 4333090152  speedup is 1.00
---------------- (adding distros)
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/xen
- creating new distro: Centos5.4-xen-x86_64
- creating new profile: Centos5.4-xen-x86_64
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot
- creating new distro: Centos5.4-x86_64
- creating new profile: Centos5.4-x86_64
- creating new profile: rescue-Centos5.4-x86_64
---------------- (associating repos)
- traversing distro Centos5.4-xen-x86_64
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/xen
- descent into /var/www/cobbler/ks_mirror/Centos5.4-x86_64
- processing repo at : /var/www/cobbler/ks_mirror/Centos5.4-x86_64
- need to process repo/comps: /var/www/cobbler/ks_mirror/Centos5.4-x86_64
- looking for /var/www/cobbler/ks_mirror/Centos5.4-x86_64/repodata/*comps*.xml
- createrepo -c cache --groupfile /var/www/cobbler/ks_mirror/Centos5.4-x86_64/repodata/comps.xml /var/www/cobbler/ks_mirror/Centos5.4-x86_64
3309/3309 - CentOS/sblim-cim-client-javadoc-1-35.el5.x86_64.rpm                 rpm64.rpm
Saving Primary metadata
Saving file lists metadata
Saving other metadata
- traversing distro Centos5.4-x86_64
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot
- descent into /var/www/cobbler/ks_mirror/Centos5.4-x86_64
- processing repo at : /var/www/cobbler/ks_mirror/Centos5.4-x86_64
- need to process repo/comps: /var/www/cobbler/ks_mirror/Centos5.4-x86_64
- looking for /var/www/cobbler/ks_mirror/Centos5.4-x86_64/repodata/*comps*.xml
- createrepo -c cache --groupfile /var/www/cobbler/ks_mirror/Centos5.4-x86_64/repodata/comps.xml /var/www/cobbler/ks_mirror/Centos5.4-x86_64
3309/3309 - CentOS/sblim-cim-client-javadoc-1-35.el5.x86_64.rpm                 rpm64.rpm
Saving Primary metadata
Saving file lists metadata
Saving other metadata
---------------- (associating kickstarts)
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/xen
- found content (breed=redhat) at /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot
[root@headnode cdimages]#


Time for cobbler configurations:- 

[root@headnode ~]# vi /etc/cobbler/settings 
server = 192.168.1.203
nextserver=192.168.1.203
pxe_just_once = 1
. . .

Now you should start the cobblerd service along httpd, dhcpd and tftp server. 


chkconfig --level 35 cobblerd on
chkconfig --level 35 httpd on
chkconfig --level 35 xinetd on
chkconfig --level 35 dhcpd on


service cobblerd restart 
service httpd restart 
service dhcpd restart 
service xinetd restart

Should enable the tftp service in /etc/xinetd.d/tftp , and change disabled=no .

Now run cobbler check and fix any obvious errors. 

cobbler check # important


You may want to disable un-necessary services on this server before moving on: 

chkconfig --level 35  NetworkManager  off                                                          
chkconfig --level 35  autofs  off                                                                  
chkconfig --level 35  avahi-daemon  off                                                            
chkconfig --level 35  avahi-dnsconfd  off                                                          
chkconfig --level 35  bluetooth  off                                                               
chkconfig --level 35  cups  off                                                                    
chkconfig --level 35  dc_client  off                                                               
chkconfig --level 35  dc_server  off                                                               
chkconfig --level 35  dhcrelay  off                                                                
chkconfig --level 35  dnsmasq  off                                                                 
chkconfig --level 35  dund  off                                                                    
chkconfig --level 35  firstboot  off                                                               
chkconfig --level 35  gpm  off                                                                     
chkconfig --level 35  hidd  off                                                                    
chkconfig --level 35  ip6tables  off                                                               
chkconfig --level 35  iptables  off                                                                
chkconfig --level 35  irda  off                                                                    
chkconfig --level 35  iscsi  off                                                                   
chkconfig --level 35  iscsid  off                                                                  
chkconfig --level 35  kdump  off                                                                   
chkconfig --level 35  libvirtd  off                                                                
chkconfig --level 35  lisa  off                                                                    
chkconfig --level 35  lvm2-monitor  off                                                            
chkconfig --level 35  mdmonitor  off                                                               
chkconfig --level 35  mdmpd  off                                                                   
chkconfig --level 35  multipathd  off                                                              
chkconfig --level 35  mysqld  off                                                                  
chkconfig --level 35  named  off                                                                   
chkconfig --level 35  netconsole  off                                                              
chkconfig --level 35  netfs  off                                                                   
chkconfig --level 35  netplugd  off                                                                
chkconfig --level 35  nfs  off                                                                     
chkconfig --level 35  nfslock  off                                                                 
chkconfig --level 35  nscd  off                                                                    
chkconfig --level 35  ntpd  off                                                                    
chkconfig --level 35  oddjobd  off                                                                 
chkconfig --level 35  pand  off                                                                    
chkconfig --level 35  pcscd  off
chkconfig --level 35  psacct  off
chkconfig --level 35  qemu  off
chkconfig --level 35  rawdevices  off
chkconfig --level 35  rdisc  off
chkconfig --level 35  readahead_later  off
chkconfig --level 35  restorecond  off
chkconfig --level 35  rpcgssd  off
chkconfig --level 35  rpcidmapd  off
chkconfig --level 35  rpcsvcgssd  off
chkconfig --level 35  saslauthd  off
chkconfig --level 35  sendmail  off
chkconfig --level 35  setroubleshoot  off
chkconfig --level 35  smartd  off
chkconfig --level 35  smb  off
chkconfig --level 35  squid  off
chkconfig --level 35  tcsd  off
chkconfig --level 35  tux  off
chkconfig --level 35  vncserver  off
chkconfig --level 35  wdaemon  off
chkconfig --level 35  winbind  off
chkconfig --level 35  wpa_supplicant  off
chkconfig --level 35  xend  off
chkconfig --level 35  xendomains  off
chkconfig --level 35  ypbind  off
chkconfig --level 35  yum-updatesd  off

End of day 1.

Day 2 

[root@headnode ~]# cobbler check
The following potential problems were detected:
#0: service dhcpd is not running
#1: since iptables may be running, ensure 69, 80, and 25151 are unblocked
#2: yum-utils need to be at least version 1.1.17 for reposync -l, current version is 1.1.16
#3: The default password used by the sample templates for newly installed machines (default_password_crypted in /etc/cobbler/settings) is still set to 'cobbler' and should be changed
[root@headnode ~]#

As you can see, 1-3 are more important to check. 4 is less important.


Check various lists from cobbler. These are created by default when you import the repository for the first time. 

[root@headnode ~]# cobbler distro list
Centos5.4-x86_64
Centos5.4-xen-x86_64
[root@headnode ~]#


[root@headnode ~]# cobbler profile list
Centos5.4-x86_64
Centos5.4-xen-x86_64
rescue-Centos5.4-x86_64
[root@headnode ~]#


Currently. No systems are registered with cobbler at the moment. As you can see below: 

[root@headnode ~]# cobbler system list
[root@headnode ~]#


First, create a (separate) profile for compute-nodes:- 

[root@headnode ~]# cobbler profile copy --name=Centos5.4-x86_64 --newname=ComputeNodes


[root@headnode ~]# cobbler profile list
Centos5.4-x86_64
Centos5.4-xen-x86_64
ComputeNodes                # <------------ Newly added profile!
rescue-Centos5.4-x86_64
[root@headnode ~]#


[root@headnode ~]# cobbler profile report --name=ComputeNodes
profile              : ComputeNodes
distro               : Centos5.4-x86_64
comment              :
created              : Tue Feb  2 09:06:50 2010
dhcp tag             : default
enable menu          : True
kernel options       : {}
kickstart            : /var/lib/cobbler/kickstarts/default.ks  # < ------------ Notice this default Kickstart.
ks metadata          : {}
mgmt classes         : []
modified             : Tue Feb  2 09:06:50 2010
name servers         : []
name servers search  : []
owners               : ['admin']
post kernel options  : {}
redhat mgmt key      : <<inherit>>
redhat mgmt server   : <<inherit>>
repos                : []
server               : <<inherit>>
template_files       : {}
virt bridge          : xenbr0
virt cpus            : 1
virt file size       : 5
virt path            :
virt ram             : 512
virt type            : qemu

[root@headnode ~]#

The name of default.ks needs to be changed, or fill default.ks with your kickstart, because it is empty by default. 

[root@headnode ~]# cat /var/lib/cobbler/kickstarts/default.ks
# this file intentionally left blank
# admins:  edit it as you like, or leave it blank for non-interactive install

There is a sample.ks, which we will use for our compute nodes. 

[root@headnode ~]# cat /var/lib/cobbler/kickstarts/sample.ks
#platform=x86, AMD64, or Intel EM64T
# System authorization information
auth  --useshadow  --enablemd5
# System bootloader configuration
bootloader --location=mbr
# Partition clearing information
clearpart --all --initlabel
# Use text mode install
text
# Firewall configuration
firewall --enabled
# Run the Setup Agent on first boot
firstboot --disable
# System keyboard
keyboard us
# System language
lang en_US
# Use network installation
url --url=$tree
# If any cobbler repo definitions were referenced in the kickstart profile, include them here.
$yum_repo_stanza
# Network information
$SNIPPET('network_config')
# Reboot after installation
reboot

#Root password
rootpw --iscrypted $default_password_crypted
# SELinux configuration
selinux --disabled
# Do not configure the X Window System
skipx
# System timezone
timezone  America/New_York
# Install OS instead of upgrade
install
# Clear the Master Boot Record
zerombr
# Allow anaconda to partition the system as needed
autopart

%pre
$SNIPPET('log_ks_pre')
$kickstart_start
$SNIPPET('pre_install_network_config')
# Enable installation monitoring
$SNIPPET('pre_anamon')

%packages
$SNIPPET('func_install_if_enabled')

%post
$SNIPPET('log_ks_post')
# Start yum configuration
$yum_config_stanza
# End yum configuration
$SNIPPET('post_install_kernel_options')
$SNIPPET('post_install_network_config')
$SNIPPET('func_register_if_enabled')
$SNIPPET('download_config_files')
$SNIPPET('koan_environment')
$SNIPPET('redhat_register')
$SNIPPET('cobbler_register')
# Enable post-install boot notification
$SNIPPET('post_anamon')
# Start final steps
$kickstart_done
# End final steps
[root@headnode ~]#

Copy the sample file as Compute.ks : 

[root@headnode ~]# cd /var/lib/cobbler/kickstarts/

[root@headnode kickstarts]# cp sample.ks Compute.ks

Edit the kickstart file Compute.ks and make necessary changes: 

[root@headnode kickstarts]# vi Compute.ks
#platform=x86, AMD64, or Intel EM64T
# System authorization information
auth  --useshadow  --enablemd5
# System bootloader configuration
bootloader --location=mbr
# Partition clearing information
clearpart --all --initlabel
# Use text mode install
text
# Firewall configuration
firewall --disabled
# Run the Setup Agent on first boot
firstboot --disable
# System keyboard
keyboard us
# System language
lang en_US
# Use network installation
url --url=$tree
# If any cobbler repo definitions were referenced in the kickstart profile, include them here.
$yum_repo_stanza
# Network information
$SNIPPET('network_config')
# Reboot after installation
reboot

#Root password
rootpw --iscrypted $default_password_crypted
# SELinux configuration
selinux --disabled
# Do not configure the X Window System
skipx
# System timezone
timezone  Asia/Riyadh
# Install OS instead of upgrade
install
# Clear the Master Boot Record
zerombr
# Allow anaconda to partition the system as needed
# autopart
part /boot --size 512
part swap --size 1024
part /    --size 4096


%pre
$SNIPPET('log_ks_pre')
$kickstart_start
$SNIPPET('pre_install_network_config')
# Enable installation monitoring
$SNIPPET('pre_anamon')

%packages
@core
$SNIPPET('func_install_if_enabled')

%post
$SNIPPET('log_ks_post')
# Start yum configuration
$yum_config_stanza
# End yum configuration
$SNIPPET('post_install_kernel_options')
$SNIPPET('post_install_network_config')
$SNIPPET('func_register_if_enabled')
$SNIPPET('download_config_files')
$SNIPPET('koan_environment')
$SNIPPET('redhat_register')
$SNIPPET('cobbler_register')
# Enable post-install boot notification
$SNIPPET('post_anamon')
# Start final steps
$kickstart_done
# End final steps
[root@headnode kickstarts]#

Now, assign the new kickstart to the profile we just created/copied: 


[root@headnode kickstarts]# cobbler profile edit --name=ComputeNodes --kickstart=/var/lib/cobbler/kickstarts/Compute.ks


Verify the change:- 

[root@headnode kickstarts]# cobbler profile report --name=ComputeNodes | grep kickstart
kickstart            : /var/lib/cobbler/kickstarts/Compute.ks
[root@headnode kickstarts]#

Define Systems

Now define/ add compute nodes. Remember, that my compute nodes have an IPMI capable interface, which have an out-of-band IP (separate network IP), than the access / production network. Right now I am only adding systems with the most basic information: 

[root@headnode ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6A:27:91
[root@headnode ~]# cobbler system add --name=compute2 --profile=ComputeNodes --mac=00:1D:09:6B:06:81


Note: In case you try to make a duplicate entry:- 

[root@headnode ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6B:06:7F
An object already exists with that name.  Try 'edit'?


Check cobbler system list: 

[root@headnode ~]# cobbler system list
compute1
compute2
[root@headnode ~]#

Looks good. You can check report of the nodes now. I am only interested in few parameters. 

[root@headnode ~]# cobbler system report | egrep -i "system|profile|power type|interface|mac"
system                : compute1
profile               : ComputeNodes
power type            : ipmitool
interface        : eth0
  mac address    : 00:1D:09:6A:27:8F
system                : compute2
profile               : ComputeNodes
power type            : ipmitool
interface        : eth0
  mac address    : 00:1D:09:6B:06:7F
[root@headnode ~]#

I need to change the interface from eth0 to eth1, because my nodes are connected to access network through eth1. Whereas eth1 is the management / IPMI / BMC network.

I can simply delete these two systems and recreate them without a hassle: 

[root@headnode ~]# cobbler system add --name=compute1 --profile=ComputeNodes --mac=00:1D:09:6A:27:91 --hostname=compute1.example.com --interface=eth1  --static=1 --ip=192.168.1.1 --subnet=255.255.255.0 --kopts=noipv6  --power-address=192.168.0.1 --power-user=root --power-pass=ipmisecret

[root@headnode ~]# cobbler system add --name=compute2 --profile=ComputeNodes --mac=00:1D:09:6B:06:81 --hostname=compute2.example.com --interface=eth1  --static=1 --ip=192.168.1.2 --subnet=255.255.255.0 --kopts=noipv6 --power-address=192.168.0.2  --power-user=root --power-pass=ipmisecret
[root@headnode ~]# 

Note: The BMC interface has the user name "root" and password "ipmisecret", are defined in each node's BMC configuration in BIOS.

Now once you modify anything in the cobbler system, you will need to do a "cobbler sync", which will make necessary changes in kickstart and tftp files and restart dhcpd. (I wasted one full day browsing the net, to figure out this one!) 

[root@headnode ~]# cobbler sync
Shutting down dhcpd:                                       [  OK  ]
Starting dhcpd:                                            [  OK  ]
[root@headnode ~]#

And if you make any changes in cobbler's /etc/cobbler/settings file or any of it's subsystem, you will need to restart cobblerd service. 

service cobblerd restart
cobbler sync


If you look at the TFTP file created by cobbler for compute1, it looks like this: 

[root@headnode ~]# cat /tftpboot/pxelinux.cfg/01-00-1d-09-6a-27-8f
default linux
prompt 0
timeout 1
label linux
        kernel /images/Centos5.4-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/Centos5.4-x86_64/initrd.img ksdevice=bootif lang=  kssendmac text  ks=http://192.168.1.200/cblr/svc/op/ks/system/compute1

[root@headnode ~]#
<pre>

The resultant dhcpd.conf file looks like this:

<pre>
cat /etc/dhcpd.conf
# ******************************************************************
# Cobbler managed dhcpd.conf file
# generated from cobbler dhcp.conf template (Tue Feb  2 11:47:35 2010)
# Do NOT make changes to /etc/dhcpd.conf. Instead, make your changes
# in /etc/cobbler/dhcp.template, as /etc/dhcpd.conf will be
# overwritten.
# ******************************************************************

ddns-update-style interim;

allow booting;
allow bootp;

ignore client-updates;
set vendorclass = option vendor-class-identifier;

subnet 192.168.1.0 netmask 255.255.255.0 {
     option routers             192.168.1.200;
     option domain-name-servers 192.168.1.200;
     option subnet-mask         255.255.255.0;
     range dynamic-bootp        192.168.1.1 192.168.1.10;
     filename                   "/pxelinux.0";
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                192.168.1.200;
}

# group for Cobbler DHCP tag: default
group {
    host generic1 {
        hardware ethernet 00:1D:09:6B:06:91;
        filename "/pxelinux.0";
        next-server 192.168.1.200;
    }
}

[root@headnode ~]#


For two nodes, I had the following TFTP files: 

[root@headnode ~]# ls  /tftpboot/pxelinux.cfg/ -l
total 12
-rw-r--r-- 1 root root  272 Feb  8 13:18 01-00-1d-09-6a-27-91
-rw-r--r-- 1 root root  272 Feb  8 13:18 01-00-1d-09-6b-06-81
-rw-r--r-- 1 root root 1091 Feb  8 13:18 default
[root@headnode ~]#


If your systems have IPMI capable interfaces, then you must make sure that you have ipmitools, ipmiconsole and ipmipower installed. 

[root@headnode ~]# yum -y install OpenIPMI-tools freeipmi-*

[root@headnode ~]# rpm -ql OpenIPMI-tools
/etc/rc.d/init.d/ipmievd
/etc/sysconfig/ipmievd
/usr/bin/ipmitool


Time to reboot these nodes and see if they are able to install. However, make sure that their netboot switch is enabled. 

[root@headnode power]# cobbler system edit --name=compute1 --netboot=1
[root@headnode power]# cobbler system edit --name=compute2 --netboot=1


[root@headnode power]# cobbler report --what=systems | egrep -i "hostname|netboot"
hostname              : compute1.example.com
netboot enabled?      : True
hostname              : compute2.example.com
netboot enabled?      : True
[root@headnode power]#

There is no "install" command. Instead all the installation mechanism has been made ready. You need to reboot the nodes to get installed. 

[root@headnode power]# cobbler system reboot --name=compute1
cobbler power configuration is:

      type   : ipmitool
      address: 192.168.0.1
      user   : root
      id     :

- /usr/bin/ipmitool -H 192.168.0.1 -U root -P ipmisecret power off
Chassis Power Control: Down/Off
cobbler power configuration is:

      type   : ipmitool
      address: 192.168.0.1
      user   : root
      id     :

- /usr/bin/ipmitool -H 192.168.0.1 -U root -P ipmisecret power on
Chassis Power Control: Up/On
[root@headnode power]#


If, after reboot of node, the installation starts and finishes and starts again, it means your nodes are stuck in an install loop. This happens, because PXE is the first boot device in the nodes, which is the required boot sequence in HPCC scenarios. What you need to do is to make sure that a particular setting is enabled in the /etc/cobblet/settings file. Without setting that, the netboot flag will remain switched on in cobbler configuration and the nodes will keep rebooting and keep installing. 

[root@headnode power]# cobbler report --what=systems | egrep -i "hostname|netboot"
hostname              : compute1.example.com
netboot enabled?      : True
hostname              : compute2.example.com
netboot enabled?      : True
[root@headnode power]#

For such situations, there is a flag in /etc/cobbler/settings, which turns off the netboot flag, when the installation is done. It is "pxe_just_once" and is set to 0 by default. It needs to be turned to 1 in this situation. 

pxe_just_once: 1

Day 3. Few other issues.

My nodes installed properly and booted properly. However they were taking too much time in trying to obtain IP for eth0 interface through dhcp. Since there is no dhcp on eth0 and I don't need to enable the eth0 interface in OS, because it is enabled and properly configured in the BIOS already, I had to find a way to fix that. Here are four items which I wanted to know the solution:

1) Static IP for eth1. Means, after installation of a node is complete, it should have a static IP configured in it instead of asking that from DHCP server. 2) Disable eth0, so it does not wait for dhcp when the node reboots from local OS/disk. 3) Console redirection during TFTP and installation. I want to see the console of the node on my headnode, instead of manually seeing it on the KVM. (It does not make sense to run to the data center everytime you need to look at screen of a node.) 4) How to check if the nodes are installed? Pretty important!


1) Static IP:-

Simply add a --static=1 to either the cobbler system add command, or modify an already defined system with this setting. 

[root@headnode ~]# cobbler system edit --name=compute1 --interface=eth1 --static=1


This is the solution for setting static IP, post-reboot. The installation got done through PXE (good).

Here is what cobbler did, after the node is installed and rebooted from local disk:- 

[root@compute1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=00:1D:09:6A:27:91
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.1.1
NETMASK=255.255.255.0
[root@compute1 ~]#

This is exactly what I wanted. Good.


2) Disable eth0. For this, I used the same technique. I just mentioned that it is also static, and did not provide any IP for it. That is because the (out of band) IP is already configured in the node's BIOS. 

[root@headnode ~]# cobbler system edit --name=compute1 --interface=eth0 --static=1

[root@headnode ~]# cobbler report system --name=compute1
system                : compute1
profile               : ComputeNodes
comment               :
. . . 
interface        : eth0
  mac address    :
  is static?     : True
  ip address     :
  subnet         :
  static routes  : []
  dhcp tag       :
interface        : eth1
  mac address    : 00:1D:09:6A:27:91
  is static?     : True
  ip address     : 192.168.1.1
  subnet         : 255.255.255.0
  static routes  : []
  dhcp tag       :

[root@headnode ~]#


Note that --delete-interface did NOT do the trick. 

[root@headnode ~]# cobbler system edit --name=compute1 --delete-interface=eth0

This did not work.


This time it worked. I have lovely results after the nodes are installed and rebooted! 

[root@compute1 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1D:09:6A:27:8F
          inet6 addr: fe80::21d:9ff:fe6a:278f/64 Scope:Link

eth1      Link encap:Ethernet  HWaddr 00:1D:09:6A:27:91
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21d:9ff:fe6a:2791/64 Scope:Link

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host

[root@compute1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:1D:09:6A:27:8F
ONBOOT=yes
BOOTPROTO=none

[root@compute1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=00:1D:09:6A:27:91
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.1.1
NETMASK=255.255.255.0
[root@compute1 ~]#

Notice that Cobbler set BOOTPROTO=none for eth0.


However the node WILL NOT ping any other node on IPMI interface. This is only controllable/ do-able in one direction, from headnode towards compute node. Remember that the headnode's management interface HAS an out of band IP configured in the OS.

Thus, don't be alarmed if you see the following, while you are logged in to any of the compute nodes: 

[root@compute1 ~]# ping 192.168.0.200
connect: Network is unreachable

[root@compute1 ~]# ping 192.168.0.201
connect: Network is unreachable
[root@compute1 ~]#

End of day 3.

Day 4. Console redirection.

For (3) We set certain parameters in the distro and profile options. 

[root@headnode ~]# cobbler distro list
Centos5.4-x86_64
Centos5.4-xen-x86_64
[root@headnode ~]#

Centos5.4-x86_64 is the distro we are using for provisioning the nodes. 

[root@headnode ~]# cobbler distro report --name=Centos5.4-x86_64
distro               : Centos5.4-x86_64
architecture         : x86_64
breed                : redhat
created              : Mon Feb  1 15:06:46 2010
comment              : rhel5.4
initrd               : /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot/initrd.img
kernel               : /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot/vmlinuz
kernel options       : {}
ks metadata          : {'tree': 'http://@@http_server@@/cblr/links/Centos5.4-x86_64'}
tree build time      : Thu Oct  1 19:57:55 2009
modified             : Mon Feb  1 15:10:12 2010
mgmt classes         : []
os version           : rhel5
owners               : ['admin']
post kernel options  : {}
redhat mgmt key      : <<inherit>>
redhat mgmt server   : <<inherit>>
template files       : {}
[root@headnode ~]#


[root@headnode ~]# cobbler distro edit --name=Centos5.4-x86_64 --kopts="console=ttyS1,115200"


[root@headnode ~]# cobbler distro report --name=Centos5.4-x86_64
distro               : Centos5.4-x86_64
architecture         : x86_64
breed                : redhat
created              : Mon Feb  1 15:06:46 2010
comment              : rhel5.4
initrd               : /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot/initrd.img
kernel               : /var/www/cobbler/ks_mirror/Centos5.4-x86_64/images/pxeboot/vmlinuz
kernel options       : {'console': 'ttyS1,115200'}
ks metadata          : {'tree': 'http://@@http_server@@/cblr/links/Centos5.4-x86_64'}
tree build time      : Thu Oct  1 19:57:55 2009
modified             : Tue Feb  9 08:24:51 2010
mgmt classes         : []
os version           : rhel5
owners               : ['admin']
post kernel options  : {}
redhat mgmt key      : <<inherit>>
redhat mgmt server   : <<inherit>>
template files       : {}

[root@headnode ~]#


Now check the profiles:- 

[root@headnode ~]# cobbler profile list
Centos5.4-x86_64
Centos5.4-xen-x86_64
ComputeNodes
rescue-Centos5.4-x86_64
[root@headnode ~]#


We are interested in profile named "ComputeNodes":- 

[root@headnode ~]# cobbler profile report --name=ComputeNodes
profile              : ComputeNodes
distro               : Centos5.4-x86_64
comment              :
created              : Tue Feb  2 09:06:50 2010
dhcp tag             : default
enable menu          : True
kernel options       : {}
kickstart            : /var/lib/cobbler/kickstarts/Compute.ks
ks metadata          : {}
mgmt classes         : []
modified             : Tue Feb  2 09:16:28 2010
name servers         : []
name servers search  : []
owners               : ['admin']
post kernel options  : {}
redhat mgmt key      : <<inherit>>
redhat mgmt server   : <<inherit>>
repos                : []
server               : <<inherit>>
template_files       : {}
virt bridge          : xenbr0
virt cpus            : 1
virt file size       : 5
virt path            :
virt ram             : 512
virt type            : qemu

[root@headnode ~]#     

[root@headnode ~]# cobbler profile edit --name=ComputeNodes --ksmeta="console=ttyS1,115200"


[root@headnode ~]# cobbler sync
Starting dhcpd:                                            [  OK  ]
[root@headnode ~]#


Try re-provisioning the node and see if it works. 

[root@headnode ~]# cobbler system edit --name=compute1 --netboot-enabled=1

[root@headnode ~]# cobbler sync
Shutting down dhcpd:                                       [  OK  ]
Starting dhcpd:                                            [  OK  ]

[root@headnode ~]# cat /tftpboot/pxelinux.cfg/01-00-1d-09-6a-27-91
default linux
prompt 0
timeout 1
label linux
        kernel /images/Centos5.4-x86_64/vmlinuz
        ipappend 2
        append initrd=/images/Centos5.4-x86_64/initrd.img ksdevice=bootif lang=  console=ttyS1,115200 text noipv6 kssendmac  ks=http://192.168.1.200/cblr/svc/op/ks/system/compute1

[root@headnode ~]#

Notice the console option in the newly generated TFTP file above.

ipmiconsole

Use the ipmiconsole command to open console of a remote node on the headnode. 

[root@headnode ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
[SOL established]

Note, if you get an error such as below, that means your headnode, does not have BMC interface correctly configured. Assign it an IP from your BMC network and bring the interface up. It should work.

The error: 

[root@headnode ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
ipmiconsole_submit_block: internal system error


The reason: 

[root@headnode ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BB
          inet6 addr: fe80::21d:9ff:fe69:f3bb/64 Scope:Link

eth1      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BD
          inet addr:192.168.1.200  Bcast:192.168.1.255  Mask:255.255.255.0

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host

The fix: 

[root@headnode ~]# ifconfig eth0 192.168.0.201 netmask 255.255.255.0 up

[root@headnode ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BB
          inet addr:192.168.0.201  Bcast:192.168.0.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr 00:1D:09:69:F3:BD
          inet addr:192.168.1.200  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21d:9ff:fe69:f3bd/64 Scope:Link

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host

[root@headnode ~]# ipmiconsole --hostname=192.168.0.1 --username=root --password=ipmisecret
[SOL established]

power on/off

Use ipmipower command to perform power operations on the nodes. 

[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --on
192.168.0.1: ok

Also if the node is off, cycle may not work:- 

[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --cycle
192.168.0.1: operation invalid

[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --off
192.168.0.1: ok

[root@headnode ~]# ipmipower --hostname=192.168.0.1 --username=root --password=ipmisecret --on
192.168.0.1: ok
[root@headnode ~]#


Problem with console entries

When using console=ttyS1,57600n8r , we get a node install time slowed down 5 times. Without it, the node install time was 1 minute. With it, the node install time reached 5+ minutes.

Using 115200 as console bnaud rate shows too much garbage on ipmiconsole as soon as the TFTP boot image is read. Before that, the BIOS display part is ok.


How to get rid of garbage on console when using 115200 baud rate?

Retrieved from "http://cooker.wbitt.com/index.php/CobblerTest"
Personal tools