Oct 232015
 

Hi, so you are here because you lost access to the management controller of your MSA2012. Not to worry, it happened to me too, after running OpenVAS security scan management interface was knocked off the network – I couldn’t access SSH let alone Web interface, couldn’t ping IP address. D’oh. But data was still being served so storage controllers were working just fine. How weird.

What I did is probably not recommended by HP or even forbidden – but some of us simply like living on the edge and experiment with stuff (and it was a good exercise to see if this kit is actually capable of failover and using multiple paths).

Besides, we have been using this particular “out of warranty” storage array for scratch data only so either short downtime or even possibility of losing it wouldn’t matter that much. But bear this in mind before you start pulling out live controller from that Payroll storage array, right? And obviously, please, do not try this with disk array equipped with a single controller only.

What you need is serial cable with mini-DB9 connector , you probably got it with your MSA2012 – the one that looks like that:

 

Hook it up to the storage array

and connect the other end to serial port on your server, preferably one that actually uses this block device (just so you can run tail -f /var/log/messages on other terminal and monitor how bad situation is).

Second element that we need is a piece of software, terminal emulator – both minicom and picocom should work and both are included in Debian and CentOS.

I used picocom, assuming your port is COM0 (because you might be using COM1 for DRAC console as described here):

yum install picocom
# or
apt-get install picocom
picocom -b 115200 /dev/ttyS0

Now, theoretically after hitting enter you should get MSA Management Controller (MC) prompt, starting with # sign. Mine was dead. This guide assumes that you cannot take it down completely and you are ready to take some risk. Also if you can, try to minimise I/O to array.

Now the the part HP won’t like, I unscrewed and removed controller B briefly, just for 10sec and then re-inserted it back into a slot. What happened, storage controller A took over operation as it should (so data access was *almost* uninterrupted) and controller B booted, bringing both management and storage services back online. With serial cable connected I was able to watch controller booting.

Once management controller B is online, wait ~2min so things settle down, then check status and if it is looking good restart controller A

show system
show configuration
restart mc A

In order to quit picocom

ctrl a 
ctrl x

 

After a short while you should have both Management controllers operational.

Tested with two HP MSA2012i (iSCSI) and one HP MSA2012sa (Direct Attach SAS) and it was smooth sailing all way down.

Now, time to find out why they went down in a first place, I suppose I’m a few firmware releases behind…


 

Sep 252015
 

 

Intro

You can find many materials on the web for this topic, this is my effort to create a receipt that actually works 🙂 Well, at least for me.

I was always struggling to get it done in “correct way” – I spent endless hours trying for example to use winbind for this, which is a mess.

Yes, there are some alternatives like FreeIPA (aka RedHat IdM), Centrify Server Suite or PowerBroker Identity Services “AD Bridge“. But for my scenario stuff already included in the distro turned out to be working just fine! All pain is over, ladies and gentlemen without further ado let me introduce you to SSSD!Simpson

Environment

Mostly Centos 6 and Centos 7 client machines + two Centos machines providing NFS and CIFS (Samba) service. Also Managed to add Ubuntu 14 LTS as a client, using same config files and Ubuntu sssd packages so nothing should be stopping you from using same method for Debian/Ubuntu machines I suppose.

Active Directory with 2003 domain functional level (I know! not my idea) and partially RFC 2307-compliant set of UNIX attributes, such as UID, UID number, GID number, login shell, etc.

To complicate matters further Linux machines are in mydomain.pl and AD domain is a subdomain addomain.mydomain.pl

 

Installing components

Here we are installing all packages and populating configuration on the client machines. Actual configuration can be obviously deployed in a various of way, using say Puppet or with some script during the machine deployment. Here I’m working on existing, standalone machine, pulling config files from internal management server using wget (or curl, whatever):

yum install sssd-krb5-common sssd-common sssd-ldap sssd-tools sssd-client sssd-ad sssd-krb5 sssd openldap-clients -y
wget http://mngm-svr.mydomain.pl/ServiceArea/sssd/sssd.conf -O /etc/sssd/sssd.conf
wget http://mngm-svr.mydomain.pl/ServiceArea/sssd/krb5.conf -O /etc/krb5.conf
wget http://mngm-svr.mydomain.pl/ServiceArea/sssd/smb.conf -O /etc/samba/smb.conf
wget http://mngm-svr.mydomain.pl/ServiceArea/sssd/access.conf -O /etc/security/access.conf

Actual config files quoted at the bottom of this post.

Configuration

Instead of hard coding ldap bind user in sssd.conf as some guides suggest I stick to using kerberos credentials embedded in keytab.

 

sed -i '/ldap_sasl_authid/d' /etc/sssd/sssd.conf
echo "ldap_sasl_authid = host/`hostname`@ADDOMAIN.MYDOMAIN.PL" >> /etc/sssd/sssd.conf 
chmod 0600 /etc/sssd/sssd.conf
rm /etc/krb5.keytab # remove anything that can possibly already be there
export KRB5_TRACE=/dev/stderr # kerberos troubleshooting! I wish I knew about it earlier
kinit dyzio # obtain ticket for user who has a right to join machine to AD domain
net ads join createupn=host/$(hostname -f)@ADDOMAIN.MYDOMAIN.PL osName="$(lsb_release -si)" osVer="$(lsb_release -sr)" createcomputer=OU=Servers,OU=Devices,DC=addomain,DC=mydomain,DC=pl -k # create account for machine using dyzio's credentials
net ads keytab create -k # create keytab
klist -ek # list entries in keytab. For client machine we are after HOST principal
kdestroy # cleanup dyzio's credenials

 

Configure PAM stack to use SSS daemon, use this

authconfig --updateall --enablesssd --enablesssdauth --ldapbasedn="dc=addomain,dc=mydomain,dc=pl" --ldapserver=""

# or 
 
authconfig --enablesssd --ldapserver=ldap://gc.addomain.mydomain.pl --ldapbasedn="ou=people,dc=addomain,dc=mydomain,dc=pl" --enablerfc2307bis --enablesssdauth --krb5kdc=dc-01.addomain.mydomain.pl --krb5realm=ADDOMAIN.MYDOMAIN.PL --disableforcelegacy --enablelocauthorize --enablemkhomedir --updateall

# nscd doesn't play nicely with sssd, lets get rid of it
service nscd stop; chkconfig nscd off
chkconfig sssd on; service sssd restart; service sssd status

That’s it, you should be good to go. Make sure your /etc/nsswitch.conf points to sssd and let sssd do the heavy lifting:

passwd: files sss
shadow: files sss
group: files sss

SSSD will use keytab to obtain TGT, lookup user account details in LDAP service in AD and perform authorisation requests using AD Kerberos service.

Cool thing is that once AD users are logged in to the client machine and have valid ticket (visible with klist) they can use this ticket to get access to other services,

ssh -k client02.mydomain #for connection to other machines) 
smbclient -k //cifs01.mydomain.pl/home # access samba shares

Bonus hint, use /etc/security/access.conf to restrict access to the box to certain users and/or groups.

Adding other SPN in AD for Linux computer object

This part is for those who wish to integrate NFS and SAMBA servers. Idea is to modify AD by adding NFS SPN to one Linux machine computer account object (NFS server account) and CIFS to other. Either login to Domain Controller and start cmd or run this on Windows client machine

runas /user:ADDOMAIN\dyzio cmd

assuming dyzio is lucky enough to be able to join machines to the domain. In terminal window that pop out, list existing and add other required spn records:

C:\Windows\system32>setspn -l cifs01
Registered ServicePrincipalNames for CN=CIFS01,OU=Servers,OU=Devices,dc=addomain,dc=mydomain,dc=pl:
HOST/cifs.mydomain.pl
HOST/CIFS01

Only HOST principal exists by default. This allows machine to authenticate users. Lets add CIFS, that will allow SAMBA to authenticate users too:

C:\Windows\system32>setspn -a CIFS/cifs01.mydomain.pl cifs01
Registering ServicePrincipalNames for CN=CIFS01,OU=Servers,OU=Devices,dc=addomain,dc=mydomain,dc=pl
 CIFS/cifs01.mydomain.pl
Updated object

Alright, now lets add NFS principal to NFS server, that should hopefully let you serve Kerberised NFS version 4.

C:\Windows\system32>setspn -a NFS/nfs01.mydomain.pl nfs01
Registering ServicePrincipalNames for CN=NFS01,OU=Servers,OU=Devices,dc=addomain,dc=mydomain,dc=pl
 NFS/nfs01.mydomain.pl
Updated object
C:\Windows\system32>setspn -l nfs01
Registered ServicePrincipalNames for CN=NFS01,OU=Servers,OU=Devices,dc=addomain,dc=mydomain,dc=pl:
 NFS/nfs01.mydomain.pl
 HOST/nfs01.mydomain.pl
 HOST/NFS01

 

Now we can go back to actual CIFS and NFS servers, fetch relevant principals and save them to each keytab.

 

kinit dyzio
net ads keytab add cifs/`hostname --long` -k
net ads keytab add nfs/`hostname --long` -k
klist -ek #can you see it?

 

NFS v4 server configuration is beyond the scope of this document. Example Samba server config at the bottom of the post. Firewall configuration for Samba in case you need it, edit /etc/sysconfig/iptables and add this before REJECT

# Allow CIFS access from everywhere
-A INPUT -p tcp -m state --state NEW -m tcp --dport 137 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 138 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 139 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 445 -j ACCEPT

 

Troubleshooting

Examine usual log files. Also, I can’t appreciate enough “export KRB5_TRACE=/dev/stderr ” – Kerberos error messages can be cryptic and misleading, bumping verbosity with KRB5_TRACE is simply priceless when tackling those issues!

tail /var/log/sssd/sssd_addomain.mydomain.pl.log
tail -n200 /var/log/sssd/ldap_child.log
# or any other files under /var/log/sssd/
# multitail is soooo good
multitail /var/log/sssd/{sssd_addomain.mydomain.pl.log,ldap_child.log} /var/log/secure /var/log/messages

export KRB5_TRACE=/dev/stderr 
net ads info -P
net ads status -P
kinit -k -t /etc/krb5.keytab host/$(hostname -f)
klist # can machine get a ticket?
net ads search "(&(objectCategory=computer)(name=*nfs01*))" -P #can machine search AD?
net ads search "(SAMAccountName=dyzio)" -P 
ldapsearch -H ldap://dc01.addomain.mydomain.pl/ -Y GSSAPI -N -b "dc=addomain,dc=mydomain,dc=pl" "(&(objectClass=user)(sAMAccountName=dyzio))" # can machine run ldap query using ticket (GSSAPI)?
ldapsearch -H ldap://dc01.addomain.mydomain.pl/ -Y GSSAPI -N -b "dc=addomain,dc=mydomain,dc=pl" "(&(objectCategory=computer)(name=*nfs01*))"
ldapsearch -H ldap://dc01.addomain.mydomain.pl/ -Y GSSAPI -N -b "dc=addomain,dc=mydomain,dc=pl" '(servicePrincipalName=cifs*)'

Config files


 

############ sssd.conf ##################
[sssd]
config_file_version = 2
domains = addomain.mydomain.pl
services = nss, pam, autofs
[nss]
[pam]
[autofs]
[domain/addomain.mydomain.pl]
debug_level = 5
id_provider = ldap
auth_provider = krb5
chpass_provider = krb5
access_provider = ldap
cache_credentials = true
ldap_referrals = false
ldap_sasl_mech = GSSAPI
ldap_schema = rfc2307bis
ldap_search_base = dc=addomain,dc=mydomain,dc=pl
ldap_user_search_base = ou=people,dc=addomain,dc=mydomain,dc=pl
ldap_group_search_base = OU=Groups,dc=addomain,dc=mydomain,dc=pl
ldap_group_object_class = group
ldap_access_order = expire
ldap_account_expire_policy = ad
ldap_force_upper_case_realm = true
ldap_user_object_class = user
ldap_user_home_directory = unixHomeDirectory
ldap_user_name = sAMAccountName
ldap_account_expire_policy = ad
ldap_sasl_authid = host/[email protected]
krb5_realm = ADDOMAIN.MYDOMAIN.PL
krb5_server = dc01.addomain.mydomain.pl
krb5_kpasswd = dc01.addomain.mydomain.pl
krb5_canonicalize = false
krb5_use_kdcinfo = false
############ /sssd.conf ##################

############ krb5.conf ##################
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_ccache_name = KEYRING:persistent:%{uid}
 default_realm = ADDOMAIN.MYDOMAIN.PL
 udp_preference_limit = 1

[realms]
 ADDOMAIN.MYDOMAIN.PL = {
 admin_server = dc01.addomain.mydomain.pl
 kdc = dc01.addomain.mydomain.pl
 kdc = dc02.addomain.mydomain.pl
 kdc = dc03.addomain.mydomain.pl
 }

[domain_realm]
 addomain.mydomain.pl = ADDOMAIN.MYDOMAIN.PL
 .addomain.mydomain.pl = ADDOMAIN.MYDOMAIN.PL
############ /krb5.conf ##################

 


############ access.conf ##################
+ : root : ALL
+ : dyzio rysio: ALL # allow these chaps
+ : kuchnia sala myjnia: ALL # allow these groups
- : ALL : ALL # middle finger to everyone else
############ /access.conf ##################

############ smb.conf - client##############
[global]
 workgroup = ADDOMAIN
 client signing = yes
 client use spnego = yes
 kerberos method = secrets and keytab
 log file = /var/log/samba/%m.log
 realm = ADDOMAIN.MYDOMAIN.PL
 security = ads
############ /smb.conf  ##################



############ smb.conf - server ############
[global]
 workgroup = ADDOMAIN
 realm = ADDOMAIN.MYDOMAIN.PL
 server string = File Server CIFS01
 interfaces = bond0
 bind interfaces only = Yes
 security = ADS
 kerberos method = secrets and keytab
 log file = /var/log/samba/samba.log
 max log size = 100
 load printers = No
 printcap name = /dev/null
 disable spoolss = Yes
 show add printer wizard = No
 idmap config ADDOMAIN : range = 10000-399990
 idmap config ADDOMAIN : backend = rid
 idmap config * : range = 10000-399990
 idmap config * : backend = tdb

[home]
 comment = Home Directories
 path = /tank/home/%U
 valid users = %U
 force user = %U
 read only = No
 create mask = 0700
 inherit permissions = Yes
 veto oplock files = /*.mdb/*.MDB/*.dbf/*.DBF/*.iso/*.pdf/*.PST/*.pst/*.tmp/*.TMP/
 locking = No
 level2 oplocks = No
############ /smb.conf  ##################