Sep 122016


This one is interesting. I’ve got a few HP BL260 blade servers, out of warranty but packed with RAM and CPU cores. Wanted to use them as compute nodes in my OpenStack cloud but all (literally all! I mean every single one!) internal SFF SATA drives died within 6 years.

Instead of replacing I decided to get rid of internal hard drives altogether and use Centos ability to use remote storage device for root partition. In the similar manner as VmWare ESXi hosts booting from iSCSI SAN – so no spinning disks inside compute node, no heat or additional energy consumption.

These cheap Blades didn’t have fancy HBA that would be able to boot from iSCSI so I used PXE booting instead. Essentially:

  • we set blade to boot from NIC
  • blade gets IP address and PXE boot server information with DHCP packet
  • blade pulls kernel and initrd from PXE server
  • blade uses iSCSI target LUN as R/W root device

iSCSI targets

iSCSI targets (one per each blade) created first on my ZFS server (NAS4FREE) – added bonus is that we can zfs-snapshot each blade’s LUN before applying critical updates.


Extents (zvols):

Name Path
mielnet-compute016 /dev/zvol/tank/mielnet-compute016
mielnet-compute017 /dev/zvol/tank/mielnet-compute017
mielnet-compute018 /dev/zvol/tank/mielnet-compute018
mielnet-compute059 /dev/zvol/tank/mielnet-compute059

Name Flags LUNs PG IG AG rw LUN0=/dev/zvol/tank/mielnet-compute016 1 1 1 rw LUN0=/dev/zvol/tank/mielnet-compute017 1 3 3 rw LUN0=/dev/zvol/tank/mielnet-compute018 1 4 4 rw LUN0=/dev/zvol/tank/mielnet-compute059 1 2 2
Initiator Groups:

Tag Initiators Networks Comment
1 ALL mielnet-compute016 Initiator Group
2 ALL mielnet-compute059 Initiator Group
3 ALL mielnet-compute017 Initiator Group
4 ALL mielnet-compute018 Initiator Group


OS installation

I used standard Centos installer, using advanced “Storage” option. Note that installation wizard failed/stuck at Grub installation phase, at this point I’ve used  installer’s second console ALT+F2 to scp kernel and initrd image out to my  PXE server.


DHCP service

We need DHCP service in order to make it working. Just standard DHCP reservations for my blades and PXE server living at address:

# cat /etc/dhcp/dhcpd.conf
deny unknown-clients;
option dhcp-max-message-size 2048;
use-host-decl-names on;
ddns-update-style none;
option domain-name "";
option domain-name-servers, ;
default-lease-time 86400;
max-lease-time 86400;
log-facility local7;
option time-servers, ;
option ntp-servers, ;

subnet netmask {
option routers ;
next-server ;
filename "pxelinux.0";
option tftp-server-name "";

host mielnet-compute016 {hardware ethernet 00:24:81:cf:xx:xx;fixed-address mielnet-compute016;}
host mielnet-compute017 {hardware ethernet 00:24:81:cf:xx:yy;fixed-address mielnet-compute017;}
host mielnet-compute018 {hardware ethernet 00:24:81:cf:xx:xy;fixed-address mielnet-compute018;}
host mielnet-compute059 {hardware ethernet 00:0c:29:02:xx:yx;fixed-address mielnet-compute059;}

PXE booting

Command gethostip will translate IP address into hexadecimal format. Then:

vim /var/lib/tftpboot/pxelinux.cfg/86977610


# cat 86977610
MENU TITLE MIELNET IT Services || Boot Server
ONTIMEOUT Centos7-mielnet-compute016

LABEL Centos7-mielnet-compute016
MENU LABEL Centos7-mielnet-compute016
kernel /images/mielnet-compute016/vmlinuz-3.10.0-327.10.1.el7.x86_64 root=/dev/sda1 ro netroot=iscsi:mielnet-compute016:[email protected] vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk LANG=en_GB.UTF-8  console=tty0 ip=enp2s0f0:dhcp  rhgb quiet
append initrd=/images/mielnet-compute016/initramfs-3.10.0-327.10.1.el7.x86_64.img

LABEL Centos7-mielnet-compute016-bridge
MENU LABEL Centos7-mielnet-compute016-bridge
kernel /images/mielnet-compute016/vmlinuz-3.10.0-327.10.1.el7.x86_64 root=/dev/sda1 ro netroot=iscsi:mielnet-compute016:[email protected] vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk LANG=en_GB.UTF-8  bridge=br-ex:enp2s0f0 ip=br-ex:dhcp console=tty0 rd.debug
append initrd=/images/mielnet-compute016/initramfs-3.10.0-327.10.1.el7.x86_64.img

LABEL Centos7-mielnet-compute016-rescue
MENU LABEL Centos7-mielnet-compute016-rescue
kernel /images/mielnet-compute016/vmlinuz-0-rescue-a8aafbe2565244fc8478818344af177d rescue vconsole.font=latarcyrheb-sun16 vconsole.keymap=uk LANG=en_GB.UTF-8 root=/dev/sda1 netroot=iscsi:mielnet-compute016:[email protected] ip=enp2s0f0:dhcp
append initrd=/images/mielnet-compute016/initramfs-0-rescue-a8aafbe2565244fc8478818344af177d.img

MENU end

make sure to replace mielnet-compute016:xxxxxxxx with your iSCSI target unique CHAP auth.

Lastly make sure we have kernel and initrd.img in place:

 # ls -l /var/lib/tftpboot/images/mielnet-compute016/
total 172068
-rw-r--r--. 1 root root   126426 Nov 19  2015 config-3.10.0-327.el7.x86_64
drwxr-xr-x. 2 root root       26 Mar 16 17:19 grub
drwx------. 3 root root       19 Mar 16 17:20 grub2
-rw-r--r--. 1 root root 41572738 Mar 16 17:21 initramfs-0-rescue-a8aafbe2565244fc8478818344af177d.img
-rw-r--r--. 1 root root 20945730 Mar 23 14:20 initramfs-3.10.0-327.10.1.el7.x86_64.img
-rw-r--r--. 1 root root 21417384 Mar 16 17:21 initramfs-3.10.0-327.el7.x86_64.img
-rw-r--r--. 1 root root 20945730 Mar 23 14:49 initramfs.img
-rw-r--r--. 1 root root 41572738 Mar 16 17:21 initramfs-rescue.img
-rw-r--r--. 1 root root   602670 Mar 16 17:20 initrd-plymouth.img
-rw-r--r--. 1 root root   252612 Nov 19  2015 symvers-3.10.0-327.el7.x86_64.gz
-rw-------. 1 root root  2963044 Nov 19  2015
-rwxr-xr-x. 1 root root  5155536 Mar 23 14:50 vmlinuz
-rwxr-xr-x. 1 root root  5156528 Mar 16 17:22 vmlinuz-0-rescue-a8aafbe2565244fc8478818344af177d
-rwxr-xr-x. 1 root root  5155536 Feb 16  2016 vmlinuz-3.10.0-327.10.1.el7.x86_64
-rwxr-xr-x. 1 root root  5156528 Nov 19  2015 vmlinuz-3.10.0-327.el7.x86_64
-rwxr-xr-x. 1 root root  5156528 Mar 16 17:22 vmlinuz-rescue

That should get you going. The only downside I can see, after upgrading Linux kernel you need to manually copy new kernel/initrd to PXE server and then change kernel filename in PXE config file manually. Fortunately, with Centos it doesn’t happen that often so I can live with that.

Apart of that, been running these Blades as compute nodes like that for a few months now with zero problems so far.

Nov 062015

ACL set codes

Working with permissions on NAS4FREE shell is pleasant and enjoyable experience …. *cough*, *cough*.

$ getfacl /tank/shared

# file: /tank/shared
# owner: aduser
# group: 500
                         [email protected]:rwxpD-a-R-c---:------:allow
                            [email protected]:rwxpD-a-R-c---:------:allow

full_set = rwxpdDaARWcCos = all permissions
modify_set = rwxpdDaARWc–s = all permissions except write_acl, write_owner
read_set = r—–a-R-c— = read_data, read_attributes, read_xattr, read_acl
write_set = -w-p—A-W—- = write_data, append_data, write_attributes, write_xattr
NFSv4 ACL legend (read from top, down and exit on first match)

                 [email protected]:--------------:-------:deny
                 [email protected]:rwxp---A-W-Co-:-------:allow
                 [email protected]:-w-p----------:-------:deny
                 [email protected]:r-x-----------:-------:allow
              [email protected]:-w-p---A-W-Co-:-------:deny
              [email protected]:r-x---a-R-c--s:-------:allow
           (r)read data +|||||||||||||:||||||+ (I)nherited
           (w)rite data -+||||||||||||:|||||+- (F)ailed access (audit)
              e(x)ecute --+|||||||||||:||||+-- (S)uccess access (audit)
               a(p)pend ---+||||||||||:|||+--- (n)o propagate
               (d)elete ----+|||||||||:||+---- (i)nherit only
         (D)elete child -----+||||||||:|+----- (d)irectory inherit
          read (a)ttrib ------+|||||||:+------ (f)ile inherit
         write (A)ttrib -------+||||||
           (R)ead xattr --------+|||||
          (W)rite xattr ---------+||||
             read a(c)l ----------+|||
            write a(C)l -----------+||
         change (o)wner ------------+|
                   sync -------------+


Changing permissions over NAS4FREE console is possible

setfacl -m u:aduser:rwxpdaARWc:fd:allow folderHere
setfacl -m g:adgroup_shared_share_access:rwxpDaARWcCo:fd:allow shared/

But it’s probably easier to manage it from Windows workstation

Pick one responsible user and grant him/her SeDiskOperatorPrivilege

net rpc rights grant aduser SeDiskOperatorPrivilege -U adadmin
net rpc rights list accounts -P
net rpc rights list -P

Also add in NAS4FREE web interface under Services|CIFS/SMB|Share|Edit|Auxiliary parameters

admin users = aduser
valid users = @adgroup_shared_share_access, @"ADgroup Shared Share Access"
write list = @adgroup_shared_share_access, @"ADgroup Shared Share Access"

And just let him do the hard work of dealing with other users.

Useful links