CENTOS Server CD project

From WBITT's Cooker!

(Difference between revisions)
Jump to: navigation, search
(References/Further Reading)
 
(9 intermediate revisions not shown)
Line 1: Line 1:
-
The next related article is: [[Custom Linux bootable CD]]
+
Please send your thoughts/comments to: kamran at wbitt dot com.
 +
 
 +
Note: This document is out dated and is superseded by the following two:
 +
 
 +
* [[CentOS 5.5 x86 64 Hypervisor Edition (Minimal)]]
 +
* [[CentOS 5.5 i386 Webserver Edition (Minimal)]]
 +
 
===Why Server CD?===
===Why Server CD?===
Line 14: Line 20:
===Creating the "core" install-tree, the easy way===
===Creating the "core" install-tree, the easy way===
-
The idea is simple. I have a CENTOS 5.4 DVD already downloaded at home (and office). To get the list of packages, I did a "core" install of CENTOS on a XEN virtual machine. (I used a kickstart file with @core mentioned in the "%packages" section. Analysing its "/root/install.log", I quickly figured out which packages got installed. (clever me!). So this is the list of packages I need from the "CentOS" directory of the install-tree. (It's about 443 packages). This resulted in an installation of about 950MB in size.  
+
The idea is simple. I have a CENTOS 5.4 DVD already downloaded at home (and office). [Note: You can also do this using just CD1 of the CENTOS/RHEL distribution]. To get the list of packages, I did a "core" install of CENTOS on a XEN virtual machine. (I used a kickstart file with @core mentioned in the "%packages" section. Analysing its "/root/install.log", I quickly figured out which packages got installed. (clever me!). So this is the list of packages I need from the "CentOS" directory of the install-tree. (It's about 443 packages). This resulted in an installation of about 950MB in size.  
Here is how I used (install.log) to generate the list of required pacakges in my repository.
Here is how I used (install.log) to generate the list of required pacakges in my repository.
Line 31: Line 37:
</pre>
</pre>
-
The contents of "installed-rpms.tmp" file are here: <b>[[Core Packages List CentOS-5.4-x86 64]]</b>
+
<pre>
 +
cat installed-rpms.tmp | cut -d: -f2 > installed-rpms.txt
 +
</pre>
 +
The contents of "installed-rpms.tmx" file are here: <b>[[Core Packages List CentOS-5.4-x86 64]]</b>
This is good enough. However, I am too much irritated by seeing bluetooth, cups, x11, and similar packages being part of the core installation. So I went one step further, and started deleting packages using yum, in a polite fashion, from this VM. This way, I was able to delete some packages, which I consider un-necessary for an installation, where only a web-server is the only desired service to run, without any backend database. (Which was the case with me.). After this surgery, I was able to bring down the disk usage to about 750 MB.  And of-course, fewer number of packages. (About 118 less rpms!). I can use that in the %post to delete the un-necessary RPMs!
This is good enough. However, I am too much irritated by seeing bluetooth, cups, x11, and similar packages being part of the core installation. So I went one step further, and started deleting packages using yum, in a polite fashion, from this VM. This way, I was able to delete some packages, which I consider un-necessary for an installation, where only a web-server is the only desired service to run, without any backend database. (Which was the case with me.). After this surgery, I was able to bring down the disk usage to about 750 MB.  And of-course, fewer number of packages. (About 118 less rpms!). I can use that in the %post to delete the un-necessary RPMs!
Line 73: Line 82:
<pre>
<pre>
-
[root@xenlab ~]# tail /tmp/minivm/installed-rpms.tmp
+
[root@xenlab ~]# tail /tmp/minivm/installed-rpms.txt
which-2.16-7.x86_64
which-2.16-7.x86_64
words-3.0-9.1.noarch
words-3.0-9.1.noarch
Line 97: Line 106:
-
[root@xenlab ~]# for file in $(cat /tmp/minivm/installed-rpms.tmp); do  
+
[root@xenlab ~]# for file in $(cat /tmp/minivm/installed-rpms.txt); do  
   cp -v /var/www/html/CentOS-5.4-x86_64/CentOS/"${file}.rpm" \  
   cp -v /var/www/html/CentOS-5.4-x86_64/CentOS/"${file}.rpm" \  
     /var/www/html/CentOS-5.4-x86_64-core/CentOS/  
     /var/www/html/CentOS-5.4-x86_64-core/CentOS/  
Line 233: Line 242:
<pre>
<pre>
-
for file in $(cat /tmp/minivm/installed-rpms.tmp); do  
+
for file in $(cat /tmp/minivm/installed-rpms.txt); do  
   wget  http://192.168.0.1/CentOS-5.4-x86_64/CentOS/${file}.rpm  \
   wget  http://192.168.0.1/CentOS-5.4-x86_64/CentOS/${file}.rpm  \
   --directory-prefix=/tmp/centos-mini/CentOS/ --no-parent  \  
   --directory-prefix=/tmp/centos-mini/CentOS/ --no-parent  \  
Line 261: Line 270:
===ToDo===
===ToDo===
-
I will make this available in few days on http://downloads.wbitt.com. (God Willing). Please note that at the moment, this is not a bootable CD image. It is a fully functional install tree, which would require you to use a kickstart with only three entries in the %packages section.
+
I will make this available in few days on http://downloads.wbitt.com. (God Willing). Please note that this is not a bootable CD, it is just an install tree, which would require you to use a kickstart with only three entries in the %packages section.
<pre>
<pre>
Line 272: Line 281:
Modifying the comps.xml and updating the repository data, so far is very exciting and geeky, BUT very painful procedure.
Modifying the comps.xml and updating the repository data, so far is very exciting and geeky, BUT very painful procedure.
-
This should be usable. I will install a VM in a couple of days and confirm that here.
+
This is usable and I have successfully installed few (real and virtual) machines with it.
-
====Modifying comps.xml====
+
P.S. The procedure to make a bootable CD out of this install tree is now complete, tested and made available here: [[Custom Linux bootable CD]]. So the bootable CD is no more under a ToDo!!!
-
comps.xml has gazillion languages represented in it.
+
-
 
+
-
<pre>
+
-
[kamran@test CentOS-5.4-x86_64-test]$ grep -v xml:lang repodata/comps.xml > /tmp/no-lang.comps.xml
+
-
 
+
-
[kamran@test CentOS-5.4-x86_64-test]$ wc -l /tmp/no-lang.comps.xml
+
-
2607 /tmp/no-lang.comps.xml
+
-
 
+
-
[kamran@test CentOS-5.4-x86_64-test]$ wc -l repodata/comps.xml
+
-
11418 repodata/comps.xml
+
-
</pre>
+
-
 
+
-
About 9000 lines removed! I wonder if it will work without any Language?!
+
===Packages which can be removed from @core===
===Packages which can be removed from @core===
Line 294: Line 290:
===References/Further Reading===
===References/Further Reading===
Some smart people have already done the bootable CD / custom CD part. Below are some links for reference:
Some smart people have already done the bootable CD / custom CD part. Below are some links for reference:
-
 
# http://people.redhat.com/rkeech/custom-distro.txt , which is now totally out-dated.  
# http://people.redhat.com/rkeech/custom-distro.txt , which is now totally out-dated.  

Current revision as of 07:08, 13 June 2011

Please send your thoughts/comments to: kamran at wbitt dot com.

Note: This document is out dated and is superseded by the following two:


Contents

Why Server CD?

Lately I thought of installing a XEN CENTOS 5.5 virtual machine on one of my servers out there. I needed the exploded tree of the ISO to be available. However, when I checked centos.org website, I noticed that they stopped distributing a "Single Server CD" after CENTOS 4. I could have downloaded a CENTOS 5.5 DVD image on the server but it was kind of un-desired / not practical for two reasons.

  • The centos.org distributes DVD iso image through bit-torrent, and my server service provider does not allow P2P or torrent traffic.
  • Even if I had one, I would have to download one big 4-5 GB file to my server. Whereas a few hundred MBs would do, if I want a text based "core" installation only.

So there arose an itch, and I decided to create a project to scratch that itch!

Objectives / Project goals

  • To create a repository / collection of files, a so called "Server CD" install-tree, small enough to provide the "core" installation, plus yum and wget.
  • To be able to create a boot-able /installable "install CD" out of the install-tree mentioned above. So that, when I (or anyone else) need to install a minimum OS on a bare-metal machine, he doesn't have to go through downloading a full DVD.

Creating the "core" install-tree, the easy way

The idea is simple. I have a CENTOS 5.4 DVD already downloaded at home (and office). [Note: You can also do this using just CD1 of the CENTOS/RHEL distribution]. To get the list of packages, I did a "core" install of CENTOS on a XEN virtual machine. (I used a kickstart file with @core mentioned in the "%packages" section. Analysing its "/root/install.log", I quickly figured out which packages got installed. (clever me!). So this is the list of packages I need from the "CentOS" directory of the install-tree. (It's about 443 packages). This resulted in an installation of about 950MB in size.

Here is how I used (install.log) to generate the list of required pacakges in my repository.

[root@xenlab minivm]# grep "Installing" install.log | awk '{print $2}'|sort > installed-rpms.tmp

Then I edited the resulting file to remove strange numbers before the package names, such as:

basesystem-8.0-5.1.1.el5.centos.noarch
cracklib-dicts-2.8.9-3.3.x86_64
1:termcap-5.5-1.20060701.1.noarch          <<<<------- Such as this one. What is this number ?
nash-5.1.19.6-54.x86_64
cat installed-rpms.tmp | cut -d: -f2 > installed-rpms.txt

The contents of "installed-rpms.tmx" file are here: Core Packages List CentOS-5.4-x86 64

This is good enough. However, I am too much irritated by seeing bluetooth, cups, x11, and similar packages being part of the core installation. So I went one step further, and started deleting packages using yum, in a polite fashion, from this VM. This way, I was able to delete some packages, which I consider un-necessary for an installation, where only a web-server is the only desired service to run, without any backend database. (Which was the case with me.). After this surgery, I was able to bring down the disk usage to about 750 MB. And of-course, fewer number of packages. (About 118 less rpms!). I can use that in the %post to delete the un-necessary RPMs!

# yum remove aspell aspell-en atk authconfig autofs avahi avahi-compat-libdns_sd \ 
  bitstream-vera-fonts bluez-gnome bluez-libs bluez-utils cairo ccid coolkey \ 
  cpuspeed crash cups cups-libs desktop-file-utils dhcpv6-client dnsmasq \ 
  dos2unix dosfstools ecryptfs-utilsed eject fbset finger firstboot-tui \ 
  fontconfig freetype GConf2 gpm gtk2 hicolor-icon-theme htmlview ifd-egate \ 
  iptables-ipv6 irda-utils irqbalance jwhois krb5-workstation ksh libdrm  \
  libICE libjpeg libnotify libpng libSM libtiff libwnck libX11 libXaulibXcursor \ 
  libXdmcp libXext libXfixes libXft libXi libXinerama libXrandr libXrender \ 
  libXres libXt libXxf86vm mailcap man man-pages mdadm mesa-libGL microcode_ctl \
  mkbootdisk mtools nano NetworkManager NetworkManager-glib newt \
  notification-daemon ntsysv numactl ORBit2 pam_ccreds pam_krb5 pam_pkcs11 \ 
  pam_smb pango paps pcmciautils pcsc-lite pcsc-lite-libs pinfo procmail rdate \ 
  redhat-lsb redhat-menus rhpl rp-pppoe rsh sendmail setuptool slang sos \ 
  specspo startup-notification syslinux system-config-network-tui \
  system-config-securitylevel-tui tcpdump trousers unix2dos vconfig \ 
  wireless-tools words wpa_supplicant xorg-x11-filesystem ypbind \ 
  yp-tools yum-updatesd

Tip: I used the following commands to generate the long yum command above:

# grep Erased /var/log/yum.log | awk '{ print $5}' | sort | uniq | tr '\n' ' '


To create my minimal repository, I first created a copy of the original install tree skipping the CentOS directory.

[root@xenlab ~]# rsync -av --exclude CentOS/  \ 
  /var/www/html/CentOS-5.4-x86_64/  /var/www/html/CentOS-5.4-x86_64-core/

Then I used the list of packages from the install.log file, and pulled the matching RPM files from the full install-tree into this "core" install-tree. I used the "installed-rpms.txt" file, I created a moment ago, above. The installed-rpms.txt file looks like this:

[root@xenlab ~]# tail /tmp/minivm/installed-rpms.txt
which-2.16-7.x86_64
words-3.0-9.1.noarch
xorg-x11-filesystem-7.1-2.fc6.noarch
yp-tools-2.9-0.1.x86_64
yum-3.2.22-20.el5.centos.noarch
yum-fastestmirror-1.1.16-13.el5.centos.noarch
yum-metadata-parser-1.1.2-3.el5.centos.x86_64
zip-2.31-2.el5.x86_64
zlib-1.2.3-3.i386
zlib-1.2.3-3.x86_64
[root@xenlab ~]#

Create the CentOS directory in the "core" install tree:

[root@xenlab ~]# mkdir /var/www/html/CentOS-5.4-x86_64-core/CentOS

Then copy the selected files from the full install-tree into this "core" tree, using the installed-rpms.txt file.



[root@xenlab ~]# for file in $(cat /tmp/minivm/installed-rpms.txt); do 
  cp -v /var/www/html/CentOS-5.4-x86_64/CentOS/"${file}.rpm" \ 
    /var/www/html/CentOS-5.4-x86_64-core/CentOS/ 
done 

Check the "core" CentOS directory:

[root@xenlab ~]# cd /var/www/html/CentOS-5.4-x86_64-core/CentOS/

[root@xenlab CentOS]# ls -l | wc -l
443

[root@xenlab CentOS]# du -sh .
273M    .
[root@xenlab CentOS]# 

Lovely! As you can see only 443 RPMs and 273 MB occupied by the RPMs only!

Note: We need to (additionally) copy the normal kernel to the new location too, because someone might want to use this for installing a non-XEN machine, or for the bare-metal machine. In that case a normal kernel would be required.

[root@xenlab ~]# cp /var/www/html/CentOS-5.4-x86_64/CentOS/kernel-2.6.18-164.el5.x86_64.rpm \ 
  /var/www/html/CentOS-5.4-x86_64-core/CentOS/ 
[root@xenlab ~]#  

Notice the difference now:

[root@xenlab CentOS]# cd /var/www/html/CentOS-5.4-x86_64-core/CentOS/

[root@xenlab CentOS]# ls | wc -l
444

[root@xenlab CentOS]# du -sh .
291M    .

The complete usage of the minimal install tree is :-

[root@xenlab ~]# du -sh /var/www/html/CentOS-5.4-x86_64-core
606M    /var/www/html/CentOS-5.4-x86_64-core
[root@xenlab ~]#  


Note: In all the steps above, we DID NOT update/change the repository (meta) data, (using createrepo command). Nor did we change anything in the installer XML files. It works just like that.

Do a test installation using this install-tree

When I was done with doing so, I created another XEN VM, and used this new (shrunk) install-tree to install the new VM to make sure it works. I used The minimal kickstart file to install that system.

It worked! Here is the newly installed system:-

[root@localhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            2.7G  986M  1.6G  39% /
tmpfs                 256M     0  256M   0% /dev/shm

[root@localhost ~]# rpm -qa | wc -l
443
[root@localhost ~]#

Then I removed the 118 packages, which I found un-necessary (as shown in the yum remove command above). The new space usage of this VM is now:

[root@localhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            2.7G  775M  1.8G  31% /
tmpfs                 256M     0  256M   0% /dev/shm
[root@localhost ~]#



Now, to use this install tree on my server on the internet, I can do one of the following:

  • Push the repository tarball to my XEN physical host/server on internet,
  • Or, make a script to "wget" all the required packages from a CENTOS mirror, create a repo for the installer, and make that available through yum on that server.

The first method is kind of immediately do-able, but not "exciting". It definitely involves copying all the install tree to the XEN host up there. To do the second method, I created a kind of script, which can pull the packages from any install tree on the internet and populate the CentOS directory on my own install-tree. Good enough, but still not geeky enough.

What I want to do now, is to be able to parse comps.xml file, present in the repodata directory of any regular install tree, find out the packages needed for "@core" and then pull those packages from the regular install tree into my own install tree. (This would be geeky/cool.)

And when I can do that, I can then create an actual bootable ISO out of it using help from many posts on the internet (mentioned at the end of this document).

Creating the "core" install-tree from a http site

Here, I am using wget to get the basic structure of CentOS full repository to my local machine, "without" pulling any rpms.

Notes:

  • In the example below, /tmp/centos-mini is the location where I am building the "core" install-tree.
  • 192.168.0.1 is a web server, with complete DVD install-tree exported through web service.
  • Interestingly CentOS directory is the "only" directory in any install tree, which contains RPM files!
This:

[root@xenlab ~]# wget --mirror  http://192.168.0.1/CentOS-5.4-x86_64/ \ 
      --directory-prefix=/tmp/centos-mini/  --no-parent \ 
      --no-host-directories --cut-dirs=1 --reject="*.rpm" \ 
      -o /tmp/wget-log

Results in:-

Saving to: `/tmp/centos-mini/CentOS/index.html?C=M;O=A'
Saving to: `/tmp/centos-mini/CentOS/index.html?C=S;O=A'
Saving to: `/tmp/centos-mini/CentOS/index.html?C=D;O=A'
Saving to: `/tmp/centos-mini/CentOS/TRANS.TBL'
Saving to: `/tmp/centos-mini/NOTES/index.html?C=N;O=D'
Saving to: `/tmp/centos-mini/NOTES/index.html?C=M;O=A'
Saving to: `/tmp/centos-mini/NOTES/index.html?C=S;O=A'
Saving to: `/tmp/centos-mini/NOTES/index.html?C=D;O=A'
Saving to: `/tmp/centos-mini/NOTES/RELEASE-NOTES-U1-as.html'
Saving to: `/tmp/centos-mini/NOTES/RELEASE-NOTES-U1-bn.html'


Good!


I also see a lot of following. Need to investigate why wget is creating these index.html files, which does not exist on the host system / web server in the first place! (This is of very low priority though):

Saving to: `/tmp/centos-mini/192.168.0.1/CentOS-5.4-x86_64/images/xen/index.html?C=S;O=D'
Saving to: `/tmp/centos-mini/192.168.0.1/CentOS-5.4-x86_64/images/xen/index.html?C=D;O=D'


Now we need to pull the required RPM files and store them manually in the CentOS directory.

for file in $(cat /tmp/minivm/installed-rpms.txt); do 
  wget  http://192.168.0.1/CentOS-5.4-x86_64/CentOS/${file}.rpm  \
  --directory-prefix=/tmp/centos-mini/CentOS/ --no-parent  \ 
  --no-host-directories --cut-dirs=1  -o /tmp/wget-log
done  

Make sure you have both kernel and kernel-xen packages in this repository. Since I already have kernel-xen, I will pull kernel-2.6.18-164.el5.x86_64.rpm :

wget  http://192.168.0.1/CentOS-5.4-x86_64/CentOS/kernel-2.6.18-164.el5.x86_64.rpm  \
  --directory-prefix=/tmp/centos-mini/CentOS/ --no-parent  \ 
  --no-host-directories --cut-dirs=1  -o /tmp/wget-log

Note: This solution (till here), has worked on a live server, using a kickstart file with only "@core" mentioned under "%packages". I am now working on the bootable ISO image, which is so far quite painful to do.

Create a tarball of your mini install-tree for easier distribution

[root@www cdimages]# tar cjf CentOS-5.4-x86_64.tar.bz2 CentOS-5.4-x86_64

[root@www cdimages]# du -sh *
537M    CentOS-5.4-x86_64
493M    CentOS-5.4-x86_64.tar.bz2
[root@www cdimages]# 

ToDo

I will make this available in few days on http://downloads.wbitt.com. (God Willing). Please note that this is not a bootable CD, it is just an install tree, which would require you to use a kickstart with only three entries in the %packages section.

%packages
@core
wget
yum

Modifying the comps.xml and updating the repository data, so far is very exciting and geeky, BUT very painful procedure.

This is usable and I have successfully installed few (real and virtual) machines with it.

P.S. The procedure to make a bootable CD out of this install tree is now complete, tested and made available here: Custom Linux bootable CD. So the bootable CD is no more under a ToDo!!!

Packages which can be removed from @core

The list is moved to separate page, here: Packages which can be removed after Core installation

References/Further Reading

Some smart people have already done the bootable CD / custom CD part. Below are some links for reference:

  1. http://people.redhat.com/rkeech/custom-distro.txt , which is now totally out-dated.
  2. http://greghaygood.com/2007/12/11/build-a-custom-centos-5-install-cd
  3. http://cinq.com/2008/04/14/rebuilding-the-dvd-iso-for-centos-5-or-rhel-5/
  4. http://www.linuxjournal.com/article/6473?page=1,0
Personal tools