Linux Storage - Removing the standby array from RDAC
Linux Storage Solution · Config document · referenced from RDAC problems and LUN removal
Time and the same file dates as the "faulty state" outputs, which suggests they were written by editing the faulty listings rather than captured after the fix; the notes do not say. The fix itself ends with a reboot and two check commands whose output the notes do not keep. The array WWNs are made-up values of the right shape.Problem 1 of my notes: mppsrv01 saw two disk arrays, the main DS5300-SITEA and the standby DS5300-SITEB, which is a mirror of the main one. The notes call that a problem above all when LVM is in use, because devlabel cannot tell the attached devices apart. This document holds the faulty state, the state that was wanted, and the command set that got there: turn off array discovery at boot, take the standby array out of the persistent mapping, remove its .wwn file, rebuild the initrd, reboot, check.
| Item | Value |
|---|---|
| Run as | root on mppsrv01, working directory /var/mpp for the ls -l, cat ./devicemapping and rm ./…wwn commands (the notes use relative paths) |
| Host | Red Hat Enterprise Linux AS 4, kernel 2.6.9-89.0.9.ELsmp, two QLogic HBAs |
| Arrays | DS5300-SITEA (ID 0, main, kept), DS5300-SITEB (ID 1, standby, removed) |
| Edits | /etc/init.d/mpp run levels (chkconfig), /var/mpp/devicemapping, /var/mpp/600a0b800012345700000000aabbcc02.wwn (removed), /boot/mpp-2.6.9-89.0.9.ELsmp.img |
| Taken | 8 September 2010 |
| Driver version | RDAC 09.03.0B05.0331 |
The listing
The faulty state, as mppUtil -a showed it:
$ mppUtil -aoutput 12 lines
Hostname = mppsrv01 Domainname = (none) Time = GMT 09/08/2010 10:50:35 --------------------------------------------------------------- Info of Array Module's seen by this Host. --------------------------------------------------------------- ID WWN Type Name --------------------------------------------------------------- 0 600a0b800012345600000000aabbcc01 FC DS5300-SITEA 1 600a0b800012345700000000aabbcc02 FC DS5300-SITEB ---------------------------------------------------------------
The faulty state in the persistent mapping:
$ cat ./devicemapping
output 2 lines
0:DS5300-SITEA 1:DS5300-SITEB
The faulty state in /var/mpp:
$ ls -l
output 3 lines
-rw------- 1 root root 361 Sep 7 11:57 600a0b800012345700000000aabbcc02.wwn -rw------- 1 root root 361 Sep 7 11:57 600a0b800012345600000000aabbcc01.wwn -rw------- 1 root root 15 Sep 8 13:26 devicemapping
The state we wanted, as mppUtil -a should show it:
$ mppUtil -aoutput 11 lines
Hostname = mppsrv01 Domainname = (none) Time = GMT 09/08/2010 10:50:35 --------------------------------------------------------------- Info of Array Module's seen by this Host. --------------------------------------------------------------- ID WWN Type Name --------------------------------------------------------------- 0 600a0b800012345600000000aabbcc01 FC DS5300-SITEA ---------------------------------------------------------------
The state we wanted in the persistent mapping:
$ cat ./devicemapping
output 1 line
0:DS5300-SITEA
The state we wanted in /var/mpp:
$ ls -l
output 2 lines
-rw------- 1 root root 361 Sep 7 11:57 600a0b800012345600000000aabbcc01.wwn -rw------- 1 root root 15 Sep 8 13:26 devicemapping
The commands
First, the attempts to discover new arrays at boot were turned off, and it was checked that mpp is off in every run level. The notes number these two lines [1] and [2]; the check has no output in the notes:
$ chkconfig mpp off $ chkconfig --list | grep mpp
Then the array DS5300-SITEB was removed from the persistent mapping:
$ mppUpdate -d DS5300-SITEBoutput 3 lines
The module DS5300-SITEB was removed from the MPP persistence mapping list (/var/mpp/devicemapping). Detected 2 QLogic Host Adapter Port(s) on the system Creating new MPP initrd image...
The WWN record of DS5300-SITEB was removed, the initrd was rebuilt once more, and the machine was restarted so that the settings take effect:
$ rm ./600a0b800012345700000000aabbcc02.wwn $ mppUpdate $ init 6
After the boot, the check that the server sees only the one array it should. The notes keep no output for these two commands; the "wanted state" listings above stand in for it:
$ mppUtil -a $ cat ./devicemapping
Reading it
| Step | Why, as far as the notes say |
|---|---|
chkconfig mpp off | /etc/init.d/mpp "discovers devices at boot"; the notes' own list of commands adds that with more than one array it is worth turning it off so that a restart causes no unwanted trouble. Without this, the next boot would find the mirror again |
chkconfig --list | the check that the service is off in every run level |
mppUpdate -d DS5300-SITEB | takes the line 1:DS5300-SITEB out of devicemapping and rebuilds the initrd, so the mapping in the initrd no longer knows the array |
rm ./…aabbcc02.wwn | mppUpdate -d does not touch the .wwn file; it had to go by hand, otherwise the driver would still hold the array's path table |
mppUpdate | the initrd is rebuilt again, now without the .wwn file; as I understand it, the .wwn files are packed into the initrd, which is why a second rebuild was needed after the rm |
init 6 | as I understand it the driver reads its persistent state only when its modules load, so a reboot is the way to apply it; the notes reboot without saying why |
mppUtil -a, cat ./devicemapping | one array, one line |
Two small things in the listings. The ls -l of the faulty state lists the …02.wwn file before the …01.wwn file, which is neither name order (plain ls -l sorts by name, and …56… comes before …57…) nor what the earlier ls -ltr in /var/mpp files printed; the lines were most likely rearranged by hand when the notes were written. And devicemapping is dated Sep 8 13:26 while the mppUtil -a above it says GMT 09/08/2010 10:50:35; the notes do not say in which order the two were taken.
What the notes do not hold: whether the standby array stayed zoned to the host on the SAN after this (the eight physical devices would then still be there, and the second problem in the same Article removes exactly those), how DS5300-SITEB came to be a mirror of DS5300-SITEA, and what happened to the LVM volume group that prompted the fix.
Checked against Red Hat Enterprise Linux 9 and 10
| As built | Today |
|---|---|
chkconfig mpp off, /etc/init.d/mpp | Red Hat's 9.7 release notes list it as a known issue that "The chkconfig package is not installed by default in RHEL 9" and point to systemctl; the package is still built (chkconfig-1.24 for 9, 1.30 for 10) |
mppUpdate rebuilding mpp-2.6.9-89.0.9.ELsmp.img, GRUB legacy menu.lst | GRUB 2 with boot loader specification entries, grubby and grub2-mkconfig; the initrd is built by dracut |
devlabel as the tool that cannot tell the mirrored devices apart | devlabel is a Red Hat Enterprise Linux 3 and 4 era tool: the package devlabel-0.48.03 ships in CentOS 3.9 and there is no devlabel package in CentOS 4.x or 5.x, so even on this host it may well have been a leftover habit rather than an installed tool. Device identity is udev's job today |
| RDAC with two arrays | Discontinued with SANtricity OS 11.25; as I see it, a mirrored array seen twice would still be a problem for LVM under DM-Multipath, and the fix would be on the SAN (zoning, LUN mapping) rather than in a driver's state files |
| Red Hat Enterprise Linux 4 | Retired 29 February 2012, extended life-cycle support ended 31 March 2017 |