How-to:warewulf debian vnfs
From SOFTICE
Synopsis
This page is about building your own Debian vnfs for warewulf using debootstrap. It's also relevant if you're trying to build a User Mode Linux root_fs file.
See also:
Building a Debian VNFS: debootstrap
Debian has a very simple way to install a chroot-ed file system. The debootstrap command is what you need so let's roll:
[master node] apt-get install debootstrap [master node] mkdir /vnfs/ [master node] mkdir /vnfs/default/ [master node] debootstrap sarge /vnfs/default/ http://ftp.us.debian.org/
Please don't use ftp.us.debian.org, I put it there as example but you should be able to use any of the various Debian mirrors listed at http://www.debian.org/mirror/list.
If you are interested in making your own Debian mirror check out how-to:install:debian_mirror
Once this is done, we can chroot inside of our vnfs and customize it:
[master node] chroot /vnfs/default/ [chroot vnfs] passwd [chroot vnfs] update-rc.d lpd remove
The apt-get sources file (/etc/apt/sources.list) can be updated and followed by a:
[chroot vnfs] apt-get update
[master node] wwvnfs --build
Also, we'll need for some of our softice scripts the /usr/bin/dialog tool so let's pack it up in the vnfs right now
[master node] cp /usr/bin/dialog /vnfs/default/usr/bin/
Fixing the hostname
Make sure that your /vnfs/default/etc/hostname configuration file is removed before to build the vnfs capsule. The hostname is passed as kernel parameter by warewulf and handled in the linuxrc script located in the nodes' wwinitrd which sources are at /var/warewulf/wwinitrd.
Using the --hybrid option
When you build the vnfs capsule on the master node, you have to use
[master node] wwvnfs --build
This builds a RAM disk which can easily become overloaded when you have a lot of things installed in your vnfs image and not many of them excluded in the master node's /etc/warewulf/exclude files.
If you need more space, you should consider using the --hybrid option (NFS mount + RAM disk)
[master node] wwvnfs --build --hybrid
This will insert in the vnfs image symbolic links to /vnfs/default/xxxx.
Update the /etc/exports on the master node to enable your cluster nodes to mount /vnfs like so;
# /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # Warewulf exports # # Do not edit below this line or it may be overwritten by wwmaster.exports! # /home 192.168.0.0/255.255.255.0(rw,root_squash,async) /vnfs 192.168.0.0/255.255.255.0(ro,no_root_squash,async)
And the cluster nodes' /etc/fstab like so to mount the /vnfs from the master node
# UNCONFIGURED FSTAB FOR BASE SYSTEM /dev/sda1 none swap sw 0 0 /dev/sda2 /tmp auto defaults 0 0 softice:/home /home nfs defaults 0 0 softice:/vnfs /vnfs nfs defaults 0 0
The last line can be replaced by
%{vnfs entry}
So that wwvnfs --build replaces it by the appropriate entry when invoked with the --hybrid option and nothing when not (Greg Kurtzer, post on warewulf mailing list).
Building a Debian VNFS: Installing arbitrary packages
The above procedure can be also applied to add arbitrary software to the nodes VNFS image, just chroot into it form the master node and make your apt-get install and other modifications of configurations (e.g. /etc/).
Here are some packages you might want to consider adding immediately:
- [chroot vnfs] apt-get install make
- [chroot vnfs] apt-get install gcc
- [chroot vnfs] apt-get install libc6-dev
The above should let you compile a kernel without problems.
- [chroot vnfs] apt-get install less
The following will help in managing your nodes configurations:
- [chroot vnfs] apt-get update
- [chroot vnfs] apt-get install dpkg
- [chroot vnfs] apt-get install rcconf
- [chroot vnfs] apt-get install perl
READ THIS Make sure that you edit the files in /etc/warewulf/vnfs/ which control what directories actually present in the vnfs capsule (/vnfs/default) will be built into what is sent to the nodes when they boot up For instance, /lib/modules/ is not included in the vnfs capsule. In this case however it is provided by the masternode itself during the boot process
Troubleshooting: apt-get problems on nodes
We have been doing our apt-get in a chroot environment. When the nodes are booted up and you are working on them you might encounter the following problem when doing an apt-get:
[cluster node] apt-get install locales E: Could not open lock file /var/lib/dpkg/lock - open (2 No such file or directory) E: Unable to lock the administration directory (/var/lib/dpkg/), are you root? [clsuter node]
Make sure that your vnfs has been rebuilt after your last apt-get update.
If this is not the source of the problem, go back to a chroot environment in your vnfs capsule (so that your changes are persistent instead of disappearing when the node reboots) and do the following:
mkdir /var/lib/dpkg mkdir /var/cache/apt mkdir /var/lib/apt mkdir /var/lib/apt/lists mkdir /var/lib/apt/lists/partial mkdir /var/cache/apt/archives mkdir /var/cache/apt/archives/partial
Then you should be able to apt-get update your system
Troubleshooting: apt-get problems in chroot with /proc
When you are using apt-get to install software while in chroot in your vnfs capsule, you might need to have /proc mounted:
[master node] cd /etc/apt [master node] apt-get install rcconf Reading Package Lists... Done Building Dependency Tree... Done rcconf is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded. 1 not fully installed or removed. Need to get 0B of archives. After unpacking 0B of additional disk space will be used. Setting up nfs-common (1.0.6-3.1) ... Starting NFS common utilities: statdstart-stop-daemon: nothing in /proc - not mounted? invoke-rc.d: initscript nfs-common, action "start" failed. dpkg: error processing nfs-common (--configure): subprocess post-installation script returned error exit status 2 Errors were encountered while processing: nfs-common E: Sub-process /usr/bin/dpkg returned an error code (1) [master node] mount proc /proc -t proc
You can then umount /proc before to exit your chroot and do a wwvnfs --build
Troubleshooting: locales problems
When using rcconf you might bump into the following problem:
[cluster node] rcconf
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Can't locate strict.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /usr/bin/rcconf line 110.
BEGIN failed--compilation aborted at /usr/bin/rcconf line 110.
chroot into your vnfs capsule and do the following:
[chroot vnfs] apt-get remove locales [chroot vnfs] apt-get install locales
Or even better:
[chroot vnfs] dpkg-reconfigure -plow locales
Then, edit /etc/environment in the chroot vnfs and fix the locale to POSIX:
LANG="POSIX"
The locales should be ok now but you still need to fix the problems with the strict.pm perl module. The perl libraries, even after doing a apt-get install of perl and cpan and then installing things with cpan in a chrooted vnfs capsule, are not there.
[master node] cd /vnfs/default [master node] cp -ar /usr/local/lib/perl usr/local/lib [master node] cp -ar /usr/local/share/perl usr/local/share [master node] cp -ar /usr/lib/perl usr/lib/ [master node] cp -ar /usr/lib/perl5 usr/lib/ [master node] cp -ar /usr/share/perl usr/share [master node] cp -ar /usr/share/perl5 usr/share
When doing so, you'll see that once booted the nodes still don't have the /usr/share/perl. Answer is that this is not included in the vnfs image and need to be explicitly configure to be included in /etc/warewulf/vnfs/
Building a Debian VNFS: Adding wulfd
Once VNFS is ok, we need to install to it the wulfd daemon so that the node can actually report to the master node on its status.
[master node] cp /etc/apt/sources.list /srv/vnfs/default/etc/apt/ [master node] cp /etc/init.d/wulfd /srv/vnfs/default/etc/init.d/ [master node] cp /usr/sbin/wulfd /srv/vnfs/default/usr/sbin/
Concerning wulfd, it's wise to make sure any firewall on the master node lets the nodes query a DNS server (UDP 53). By default, debootstrat copied the /etc/resolv.conf from the master node to the vnfs thus having the nodes query an external name server. You might want to just let them do that or install your own DNS or DNS cache on the master node. This might eventually may be described in how-to:install:DNS.
Back to our warewulf cluster. Now we can chroot inside our Debian VNFS and finish our install after making sure we have a user root entry in /etc/passwd.
[chroot vnfs] mkdir /var/lock/subsys
without the above, starting wulfd will cause a "Couldn't write lock file /var/lock/subsys/wulfd!..."
We also need to create a /etc/warewulf/sysconfig/wulfd file containing:
# Define the master node that wulfd should be sending the node status to ADMIN_MASTER='localhost'
Since /etc/init.d/wulfd starts by testing its existence and will fail to proceed if it isn't there to indicate which masternode to report to. Also, yes the localhost in the above is correct and will be expanded into softice-admin in the vnfs when the nodes boot it up.
After you installed all other Debian packages you might need on the nodes, configure the init scripts with:
[chroot vnfs] rcconf
Select wulfd and click ok in the curses interface that pops up.
Once everything is taken care of, let's rebuild the vnfs image /srv/vnfs/default/vnfs.tar.gz
[master node] wwvnfs --build
At this point, you can reboot your node, it'll run wulfd on boot and therefore be visible from the master node by typing:
[master node] wwlist
Building a Debian VNFS: SSH
Finally you might want to replace rsh by ssh inside of the cluster:
[master node] su - [master node] chroot /vnfs/default/ [chroot vnfs] apt-get install ssh [chroot vnfs] select SSH 2 protocol only [chroot vnfs] select /usr/lib/ssh-keysign to be suid root [chroot vnfs] accept to run the sshd server
Then we follow the steps indicated on the warewulf wiki (Reference is http://warewulf.lbl.gov/pmwiki/index.php?n=Documentation.Customization) with some Debian-specific alterations:
[chroot vnfs] ssh-keygen -t dsa
- since we did that in the chroot'ed environment no need to copy anything into it by hand
We need to generate again the vnfs:
[master node] wwvnfs --build
Lastly, we will modify /etc/warewulf/master.conf to indicate we choose ssh over rsh:
[master node] vi /etc/warewulf/master.conf
Replace:
rsh command = /usr/bin/rsh
by:
rsh command = /usr/bin/ssh
Building a Debian VNFS: passwords & users
Joe Stanaback wrote a script available at Warewulf update nodes users accounts script which helps in picking out of /etc/passwd accounts that will be propagated to the warewulf configuration files in /etc/warewulf/.
When you run this script, make a copy of your configuration as follows:
cp /etc/warewulf/nodes/nodegroup1/config /etc/warewulf/nodes.conf
Now you can run the script which will make a nodes.conf.old backup copy and update nodes.conf
/usr/local/bin/update_cluster.pl
Check nodes.conf then move it to its correct location:
cp /etc/warewulf/nodes.conf /etc/warewulf/nodegroup1/config
Initially make sure you have the basic entries / passwords:
- cp /etc/passwd /vnfs/default/etc/
- cp /etc/shadow* /vnfs/default/etc/
- cp /etc/group* /vnfs/default/etc/
Building a Debian VNFS: adding SOFTICE thingies
First thing first, in order to be able to play with User Mode Linux on the cluster nodes we need to once again chroot into /vnfs/default and apt-get there kernel sources:
- su -
- chroot /vnfs/default/
- wget ftp://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.13.tar.gz
apt-get won't be of much use here since we want to use one of the latest available kernel source tree in order to skip over the rather buggy period for UML on kernels 2.6.8 to 2.6.11. For similar reasons, the master nodes and cluster nodes kernels also have similar constraints.
- cp linux-2.6.13.tar.gz /usr/src
- cd /usr/src
- tar oxzf linux-2.6.13.tar.gz
- ln -s linux-2.6.13 linux
At this point the kernel sources are there and ready to be compiled. By default debootstrap won't install development tools so we need to ensure we have compiler, make, ...
- apt-get install make
- in the above, accept to update libc6
- apt-get install gcc
- apt-get install libc6-dev
- apt-get install vim
Once this is done, we can build our User Mode Linux virtual machine:
- cd /usr/src/linux
- make ARCH=um defconfig
- make ARCH=um linux
just to make sure, let's test uml...
- cp SOMEWHERE/debian.3.0r2.fs /usr/src
- cp /usr/src/linux/linux /usr/src/linux-uml
- cd /usr/src
- ./linux-uml con=null con1=fd:0,fd:1 ubd0=debian.3.0r2.fs
Building a Debian VNFS: NFS mount of /home from masternode
Make sure the master node is exporting /home to the cluster lan
# /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # Warewulf exports # # Do not edit below this line or it may be overwritten by wwmaster.exports! # /home 192.168.0.0/255.255.255.0(rw,root_squash,async) /vnfs 192.168.0.0/255.255.255.0(ro,no_root_squash,async) softice:~# 0 bash
Then, restart the nfs service;
/etc/init.d/nfs-common restart /etc/init.d/nfs-user-server restart
On the node vnfs, make sure the /etc/fstab file mounts /home through NFS;
# UNCONFIGURED FSTAB FOR BASE SYSTEM /dev/sda1 none swap sw 0 0 /dev/sda2 /tmp auto defaults 0 0 softice:/home /home nfs defaults 0 0
Building a Debian VNFS: Configure Warewulf with SMB
A bug in NFS prevents prevents UML (User Mode Linux) from functioning properly when accessing files stored on NFS filesystems. The How-to: Configure Warewulf with SMB document explains how the SOFTICE project uses SMB filesystems to bypass this problem.
References
- Warewulf website
http://warewulf.lbl.gov/pmwiki/index.php?n=Main.HomePage
- Caosity website (warewulf email lists)
- Debian New Maintainers' Guide
http://www.debian.org/doc/maint-guide/
- Debian Binary Package Building HOWTO
http://www.tldp.org/HOWTO/Debian-Binary-Package-Building-HOWTO/index.html
- Link page from the above document
http://www.tldp.org/HOWTO/Debian-Binary-Package-Building-HOWTO/x267.html
- debian-mentors email list
http://lists.debian.org/debian-mentors/
- Debian Policy Manual
http://www.debian.org/doc/debian-policy/
- Shadow Passwords How-To
http://www.ibiblio.org/pub/Linux/docs/HOWTO/Shadow-Password-HOWTO
