I think Karl configured the original RAID array using the bios-like "ORCA" utility, which you get access to by hitting F8 at the right time. I think we get more functionality from "hpacucli".
Do stuff like:
sudo /usr/sbin/hpacucli ctrl all show config detail sudo /usr/sbin/hpacucli controller slot=1 physicaldrive allunassigned show sudo /usr/sbin/hpacucli controller slot=1 physicaldrive all show
Used the following commands to add a logical volume. The first line creates an array with the name "B", since "A" already existed:
sudo /usr/sbin/hpacucli ctrl slot=1 create type=ld drives=2e:1:1-2e:1:24 raid=adg sudo /usr/sbin/hpacucli ctrl slot=1 array B add spares=2e:1:25 sudo /usr/sbin/hpacucli ctrl slot=1 show config
After this (and potentially, a reboot) can see the disk with:
cat /proc/driver/cciss/cciss2
Ole has a wrapper script to simplify using hpacucli in /nfs/slac/g/suncatfs/sw/package/smartshow
options
Usage: "sudo /u1/sw/package/smartshow" with flags one of -c -d -e -1 -a -l -D
-c shows controller status
-d show disk
-1 show disk compact (1 line per)
-e show enclosure
-l logical drives
location, status, size, model number, firmware rev
1I:1:1: OK: 500 GB: HP MM0500FAMYT : HPD3
1I:1:2: OK: 500 GB: HP MM0500FAMYT : HPD3
2E:1:1: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:2: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:3: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:4: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:5: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:6: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:7: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:8: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:9: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:10: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:11: OK: 600 GB: HP EF0600FARNA : HPD2
2E:1:12: OK: 600 GB: HP EF0600FARNA : HPD2
How Karl Monitors the NFS RAID Status
There is a cronjob on suncatfs1 which checks the status of the raid cards every two hours. The current status is updated in this file:
$ cat /afs/slac/g/scs/systems/system.info/suncatfs1/cciss_vol_status /dev/cciss/c0d0: (Smart Array P410i) RAID 1 Volume 0 status: OK. /dev/cciss/c2d0: (Smart Array P812) RAID 6 Volume 0 status: OK. At least one spare drive designated. At least one spare drive remains available. /dev/cciss/c2d0: (Smart Array P812) Enclosure D2600 SAS AJ940A (S/N: CN894700M7 ) on Bus 2, Physical Port 2E status: OK.
There is a similar directory in the same path for suncatfs2.
cciss is a "block driver for older hp smart array raid controllers. If there are any changes to the status (if the output is different at all), you and I will get an email.
The internal OS disks are the RAID1. The NFS disks are the RAID 6.
Configuring the RAID Filesystem
From Karl A.:
# -------------------------------------------------------------------------- # HP Smart External Array P812 # 12 600 GB, bays 1-11 RAID 6, bay 12 is hot spare # RAID configured in BIOS setup utility (F8 at the right time). -ksa Oct 2010 # -------------------------------------------------------------------------- # LVM/ext3 filesystem. -ksa Oct 2010 # # parted /dev/cciss/c2d0 # mkpart primary ext2 # Start? 1 # End? 100% # set 1 lvm on # pvcreate /dev/cciss/c2d0p1 # vgcreate VG1 /dev/cciss/c2d0p1 # lvcreate -n LV1.u1 -L 4.5T VG1 # mke2fs -v -j -m1 /dev/VG1/LV1.u1 # lvextend /dev/VG1/LV1.u1 /dev/cciss/c2d0p1 # e2fsck -f /dev/VG1/LV1.u1 # resize2fs /dev/VG1/LV1.u1
For adding the second tray, Karl and I did:
# 2nd Array # parted /dev/cciss/c2d1 # mklabel # gpt # mkpart primary ext2 # Start? 1 # End? 100% # set 1 lvm on # quit # # pvcreate /dev/cciss/c2d1p1 # vgextend VG1 /dev/cciss/c2d1p1 # lvextend /dev/VG1/LV1.u1 /dev/cciss/c2d1p1 # Extending logical volume LV1.u1 to 16.92 TB # Logical volume LV1.u1 successfully resized # # # we then tried this command: # resize2fs /dev/VG1/LV1.u1 # but 16.92TB was too big for ext3 filesystem resize2fs, we got the error: # File too large while trying to determine filesystem size # so then we reduced the size of the logical volume to just under 16TB # with this command: # lvreduce -L -1t /dev/VG1/LV1.u1 # after that, the resize2fs command worked: # # resize2fs /dev/VG1/LV1.u1
The only command that took any time was resize2fs, which took about 2 hours for the extra 10TB. Even the hpacucli commands were quick.
Smartshow output
Nov. 22 2011 here
Dec. 7, 2011 here (after firmware upgrade)