Intro
This one is interesting. I’ve got a few HP BL260 blade servers, out of warranty but packed with RAM and CPU cores. Wanted to use them as compute nodes in my OpenStack cloud but all (literally all! I mean every single one!) internal SFF SATA drives died within 6 years.
Instead of replacing I decided to get rid of internal hard drives altogether and use Centos ability to use remote storage device for root partition. In the similar manner as VmWare ESXi hosts booting from iSCSI SAN – so no spinning disks inside compute node, no heat or additional energy consumption.
These cheap Blades didn’t have fancy HBA that would be able to boot from iSCSI so I used PXE booting instead. Essentially:
- we set blade to boot from NIC
- blade gets IP address and PXE boot server information with DHCP packet
- blade pulls kernel and initrd from PXE server
- blade uses iSCSI target LUN as R/W root device
iSCSI targets
iSCSI targets (one per each blade) created first on my ZFS server (NAS4FREE) – added bonus is that we can zfs-snapshot each blade’s LUN before applying critical updates.
Extents (zvols): Name Path mielnet-compute016 /dev/zvol/tank/mielnet-compute016 mielnet-compute017 /dev/zvol/tank/mielnet-compute017 mielnet-compute018 /dev/zvol/tank/mielnet-compute018 mielnet-compute059 /dev/zvol/tank/mielnet-compute059 Targets: Name Flags LUNs PG IG AG iqn.2007-09.jp.ne.peach.istgt:mielnet-compute016 rw LUN0=/dev/zvol/tank/mielnet-compute016 1 1 1 iqn.2007-09.jp.ne.peach.istgt:mielnet-compute017 rw LUN0=/dev/zvol/tank/mielnet-compute017 1 3 3 iqn.2007-09.jp.ne.peach.istgt:mielnet-compute018 rw LUN0=/dev/zvol/tank/mielnet-compute018 1 4 4 iqn.2007-09.jp.ne.peach.istgt:mielnet-compute059 rw LUN0=/dev/zvol/tank/mielnet-compute059 1 2 2 Initiator Groups: Tag Initiators Networks Comment 1 ALL 10.10.100.16/32 mielnet-compute016 Initiator Group 2 ALL 10.10.100.59/32 mielnet-compute059 Initiator Group 3 ALL 10.10.100.17/32 mielnet-compute017 Initiator Group 4 ALL 10.10.100.18/32 mielnet-compute018 Initiator Group
OS installation
I used standard Centos installer, using advanced “Storage” option. Note that installation wizard failed/stuck at Grub installation phase, at this point I’ve used installer’s second console ALT+F2 to scp kernel and initrd image out to my PXE server.
DHCP service
We need DHCP service in order to make it working. Just standard DHCP reservations for my blades and PXE server living at 10.10.100.57 address:
# cat /etc/dhcp/dhcpd.conf
#########################
deny unknown-clients;
authoritative;
option dhcp-max-message-size 2048;
use-host-decl-names on;
ddns-update-style none;
option domain-name "mielnet.pl";
option domain-name-servers 8.8.8.8, 8.8.4.4 ;
default-lease-time 86400;
max-lease-time 86400;
log-facility local7;
option time-servers ntp0.mielnet.pl,inti.mielnet.pl ;
option ntp-servers ntp0.mielnet.pl,inti.mielnet.pl ;
#########################
subnet 10.10.100.0 netmask 255.255.255.0 {
option routers 10.10.100.254 ;
next-server 10.10.100.57 ;
filename "pxelinux.0";
option tftp-server-name "10.10.100.57";
}
host mielnet-compute016 {hardware ethernet 00:24:81:cf:xx:xx;fixed-address mielnet-compute016;}
host mielnet-compute017 {hardware ethernet 00:24:81:cf:xx:yy;fixed-address mielnet-compute017;}
host mielnet-compute018 {hardware ethernet 00:24:81:cf:xx:xy;fixed-address mielnet-compute018;}
host mielnet-compute059 {hardware ethernet 00:0c:29:02:xx:yx;fixed-address mielnet-compute059;}
PXE booting
Command gethostip 10.10.100.16 will translate IP address into hexadecimal format. Then:
vim /var/lib/tftpboot/pxelinux.cfg/86977610
# cat 86977610 DEFAULT menu PROMPT 0 MENU TITLE MIELNET IT Services || Boot Server TIMEOUT 20 TOTALTIMEOUT 200 ONTIMEOUT Centos7-mielnet-compute016 LABEL Centos7-mielnet-compute016 MENU LABEL Centos7-mielnet-compute016 kernel /images/mielnet-compute016/vmlinuz-3.10.0-327.10.1.el7.x86_64 root=/dev/sda1 ro netroot=iscsi:mielnet-compute016:[email protected]::3260::iqn.2007-09.jp.ne.peach.istgt:mielnet-compute016 rd.iscsi.initiator=iqn.1994-05.com.redhat:4b7c6d70242b vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk LANG=en_GB.UTF-8 console=tty0 ip=enp2s0f0:dhcp rhgb quiet append initrd=/images/mielnet-compute016/initramfs-3.10.0-327.10.1.el7.x86_64.img LABEL Centos7-mielnet-compute016-bridge MENU LABEL Centos7-mielnet-compute016-bridge kernel /images/mielnet-compute016/vmlinuz-3.10.0-327.10.1.el7.x86_64 root=/dev/sda1 ro netroot=iscsi:mielnet-compute016:[email protected]::3260::iqn.2007-09.jp.ne.peach.istgt:mielnet-compute016 rd.iscsi.initiator=iqn.1994-05.com.redhat:4b7c6d70242b vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk LANG=en_GB.UTF-8 bridge=br-ex:enp2s0f0 ip=br-ex:dhcp console=tty0 rd.shell rd.debug append initrd=/images/mielnet-compute016/initramfs-3.10.0-327.10.1.el7.x86_64.img LABEL Centos7-mielnet-compute016-rescue MENU LABEL Centos7-mielnet-compute016-rescue kernel /images/mielnet-compute016/vmlinuz-0-rescue-a8aafbe2565244fc8478818344af177d rescue vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk LANG=en_GB.UTF-8 root=/dev/sda1 netroot=iscsi:mielnet-compute016:[email protected]::3260::iqn.2007-09.jp.ne.peach.istgt:mielnet-compute016 ip=enp2s0f0:dhcp rd.iscsi.initiator=iqn.1994-05.com.redhat:4b7c6d70242b append initrd=/images/mielnet-compute016/initramfs-0-rescue-a8aafbe2565244fc8478818344af177d.img MENU end
make sure to replace mielnet-compute016:xxxxxxxx with your iSCSI target unique CHAP auth.
Lastly make sure we have kernel and initrd.img in place:
# ls -l /var/lib/tftpboot/images/mielnet-compute016/ total 172068 -rw-r--r--. 1 root root 126426 Nov 19 2015 config-3.10.0-327.el7.x86_64 drwxr-xr-x. 2 root root 26 Mar 16 17:19 grub drwx------. 3 root root 19 Mar 16 17:20 grub2 -rw-r--r--. 1 root root 41572738 Mar 16 17:21 initramfs-0-rescue-a8aafbe2565244fc8478818344af177d.img -rw-r--r--. 1 root root 20945730 Mar 23 14:20 initramfs-3.10.0-327.10.1.el7.x86_64.img -rw-r--r--. 1 root root 21417384 Mar 16 17:21 initramfs-3.10.0-327.el7.x86_64.img -rw-r--r--. 1 root root 20945730 Mar 23 14:49 initramfs.img -rw-r--r--. 1 root root 41572738 Mar 16 17:21 initramfs-rescue.img -rw-r--r--. 1 root root 602670 Mar 16 17:20 initrd-plymouth.img -rw-r--r--. 1 root root 252612 Nov 19 2015 symvers-3.10.0-327.el7.x86_64.gz -rw-------. 1 root root 2963044 Nov 19 2015 System.map-3.10.0-327.el7.x86_64 -rwxr-xr-x. 1 root root 5155536 Mar 23 14:50 vmlinuz -rwxr-xr-x. 1 root root 5156528 Mar 16 17:22 vmlinuz-0-rescue-a8aafbe2565244fc8478818344af177d -rwxr-xr-x. 1 root root 5155536 Feb 16 2016 vmlinuz-3.10.0-327.10.1.el7.x86_64 -rwxr-xr-x. 1 root root 5156528 Nov 19 2015 vmlinuz-3.10.0-327.el7.x86_64 -rwxr-xr-x. 1 root root 5156528 Mar 16 17:22 vmlinuz-rescue
That should get you going. The only downside I can see, after upgrading Linux kernel you need to manually copy new kernel/initrd to PXE server and then change kernel filename in PXE config file manually. Fortunately, with Centos it doesn’t happen that often so I can live with that.
Apart of that, been running these Blades as compute nodes like that for a few months now with zero problems so far.