Restarting MSA2012 management controller using serial cable

Hi, so you are here because you lost access to the management controller of your MSA2012. Not to worry, it happened to me too, after running OpenVAS security scan management interface was knocked off the network – I couldn’t access SSH let alone Web interface, couldn’t ping IP address. D’oh. But data was still being served so storage controllers were working just fine. How weird.

What I did is probably not recommended by HP or even forbidden – but some of us simply like living on the edge and experiment with stuff (and it was a good exercise to see if this kit is actually capable of failover and using multiple paths).

Besides, we have been using this particular “out of warranty” storage array for scratch data only so either short downtime or even possibility of losing it wouldn’t matter that much. But bear this in mind before you start pulling out live controller from that Payroll storage array, right? And obviously, please, do not try this with disk array equipped with a single controller only.

What you need is serial cable with mini-DB9 connector , you probably got it with your MSA2012 – the one that looks like that:

 

Hook it up to the storage array

and connect the other end to serial port on your server, preferably one that actually uses this block device (just so you can run tail -f /var/log/messages on other terminal and monitor how bad situation is).

Second element that we need is a piece of software, terminal emulator – both minicom and picocom should work and both are included in Debian and CentOS.

I used picocom, assuming your port is COM0 (because you might be using COM1 for DRAC console as described here):

yum install picocom
# or
apt-get install picocom
picocom -b 115200 /dev/ttyS0

Now, theoretically after hitting enter you should get MSA Management Controller (MC) prompt, starting with # sign. Mine was dead. This guide assumes that you cannot take it down completely and you are ready to take some risk. Also if you can, try to minimise I/O to array.

Now the the part HP won’t like, I unscrewed and removed controller B briefly, just for 10sec and then re-inserted it back into a slot. What happened, storage controller A took over operation as it should (so data access was *almost* uninterrupted) and controller B booted, bringing both management and storage services back online. With serial cable connected I was able to watch controller booting.

Once management controller B is online, wait ~2min so things settle down, then check status and if it is looking good restart controller A

show system
show configuration
restart mc A

In order to quit picocom

ctrl a 
ctrl x

 

After a short while you should have both Management controllers operational.

Tested with two HP MSA2012i (iSCSI) and one HP MSA2012sa (Direct Attach SAS) and it was smooth sailing all way down.

Now, time to find out why they went down in a first place, I suppose I’m a few firmware releases behind…

---