# Old DL380 Raid Notes ## Problem: Where are my disks??? When we installed the os on our new (to us) prolient DL380, Only a single disk was visible in spite of there having been 6 disks installed. This is because the DL380s disk controller was set up in raid mode an did not expose disks until they were configured as "logical" disks. This is unlike the Dell PowerEdge we have which detects and presents the drives in a hot swappable fashion while still allowing some disks to participate in raid arrays. Since we use hardware raid mirroring on the boot disks, Adding, removing or replacing disks requires configuring the raid controller. ## Configuring the disks using the raid controller bios _... use some words here ..._ ```sh steve:~ don$ ssh -p 22222 feurig@vpn.suspectdevices.com User:feurig logged-in to kb2018.suspectdevices.com(192.168.31.119 / FE80::9E8E:99FF:FE0C:BAD8) iLO 3 Advanced for BladeSystem 1.88 at Jul 13 2016 Server Name: kb2018 Server Power: On hpiLO-> vsp Virtual Serial Port Active: COM2 Starting virtual serial port. Press 'ESC (' to return to the CLI Session. root@kb2018:~# fdisk -l|grep Disk\ \/ Disk /dev/loop0: 86.9 MiB, 91099136 bytes, 177928 sectors Disk /dev/loop1: 87.9 MiB, 92164096 bytes, 180008 sectors Disk /dev/loop2: 63.4 MiB, 66486272 bytes, 129856 sectors Disk /dev/sda: 136.7 GiB, 146778685440 bytes, 286677120 sectors root@kb2018:~# reboot [ OK ] Stopped Stop ureadahead data collection 45s after completed Stopping Session 98 of user feurig. Stopping Availability of block devices... ... [ OK ] Reached target Shutdown. [ OK ] Reached target Final Step. Starting Reboot... [292357.910620] reboot: Restarting system ``` After several seconds you will see a text based bios screen [[Image(CaptiveRaidController:ILo3SSHConsoleBooting.png)]] After the network controller is started the raid controller will give you a chance to configure it. _PRESS F8 NOW!! _ \\ [[Image(CaptiveRaidController:PressF8NOW.png)]] If you miss it you will have to escape back to the ILO3 and power cycle the machine. _(This is ok because the disks are not active until the machine actually boots)_ ```sh Booting from Hard Drive C: ( hpiLO-> power off hard status=0 status_tag=COMMAND COMPLETED Wed Sep 26 15:31:57 2018 Forcing server power off ....... Please wait 6 seconds for this operation to complete. hpiLO-> power status=0 status_tag=COMMAND COMPLETED Wed Sep 26 15:32:04 2018 power: server power is currently: Off hpiLO-> power on status=0 status_tag=COMMAND COMPLETED Wed Sep 26 15:32:21 2018 Server powering on ....... hpiLO-> vsp Virtual Serial Port Active: COM2 Starting virtual serial port. Press 'ESC (' to return to the CLI Session. ``` Once in the raid controller bios you will get a main menu. [[Image(CaptiveRaidController:ViewLogicalDrive.png)]] If you select view logical drives will see that the first two disks are combined into a mirrored pair and that there are no other drives defined. So we select "Create Logical Drive". Which gives us the following screen. [[Image(CaptiveRaidController:CreateLogicalDriveDefaults.png)]] Notice that the defaults are to create a raid 1+0 array with the first two matching disks. Deselecting either disk (down arrow, spacebar) will cause the raid configuration to automatically drop to RAID 0 Press Enter when finished. The next screen will ask you to verify the creation Repeat this for each remaining disk. When you are finished you can view the logical drives. [[Image(CaptiveRaidController:RaidConfFinished.png)]] The key will walk you back out so you can continue to boot. ## success ```sh root@kb2018:~# fdisk -l|grep Disk\ \/ Disk /dev/loop0: 86.9 MiB, 91099136 bytes, 177928 sectors Disk /dev/loop1: 87.9 MiB, 92164096 bytes, 180008 sectors Disk /dev/loop2: 63.4 MiB, 66486272 bytes, 129856 sectors Disk /dev/sda: 136.7 GiB, 146778685440 bytes, 286677120 sectors Disk /dev/sdb: 223.6 GiB, 240021504000 bytes, 468792000 sectors Disk /dev/sdc: 223.6 GiB, 240021504000 bytes, 468792000 sectors Disk /dev/sdd: 279.4 GiB, 299966445568 bytes, 585871964 sectors Disk /dev/sde: 279.4 GiB, 299966445568 bytes, 585871964 sectors root@kb2018:~# ``` ## Using HP utilities to configure the controller without downing the server HP provides utilities and officially supports bionic and hosts a repo for it. It includes a server that can be accessed graphically as well as a command line interface. [#fn1 (1)] For the purpose of maintaining disks we only need ssacli and perhaps ssaducli. * install the hp supported utilities. ```sh root@kb2018:~# echo "deb http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current non-free" >> /etc/apt/sources.list.d/hp.list root@kb2018:~# root@kb2018:/etc/apt# wget http://downloads.linux.hpe.com/SDR/repo/mcp/GPG-KEY-mcp --2018-11-12 09:00:29-- http://downloads.linux.hpe.com/SDR/repo/mcp/GPG-KEY-mcp Resolving downloads.linux.hpe.com (downloads.linux.hpe.com)... 15.249.152.85 Connecting to downloads.linux.hpe.com (downloads.linux.hpe.com)|15.249.152.85|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 994 Saving to: ‘GPG-KEY-mcp’ GPG-KEY-mcp 100%[=============================================================================================>] 994 --.-KB/s in 0s 2018-11-12 09:00:30 (90.5 MB/s) - ‘GPG-KEY-mcp’ saved [994/994] root@kb2018:/etc/apt# apt-key add GPG-KEY-mcp OK root@kb2018:/etc/apt# apt-get update Ign:1 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current InRelease Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB] Hit:3 http://us.archive.ubuntu.com/ubuntu bionic InRelease Get:4 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release [6,051 B] Get:5 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release.gpg [490 B] Get:6 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] Hit:7 http://archive.ubuntu.com/ubuntu bionic InRelease Ign:5 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release.gpg Get:8 http://archive.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB] Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] Reading package lists... Done W: GPG error: http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY C208ADDE26C2B797 E: The repository 'http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release' is not signed. N: Updating from such a repository can't be done securely, and is therefore disabled by default. N: See apt-secure(8) manpage for repository creation and user configuration details. root@kb2018:/etc/apt# key=C208ADDE26C2B797 root@kb2018:/etc/apt# gpg --keyserver keyserver.ubuntu.com --recv-keys $key gpg: key C208ADDE26C2B797: public key "Hewlett Packard Enterprise Company RSA-2048-25 " imported gpg: Total number processed: 1 gpg: imported: 1 root@kb2018:/etc/apt# gpg --armor --export $key |apt-key add - OK root@kb2018:/etc/apt# apt-get update Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB] Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease Ign:5 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current InRelease Get:6 http://archive.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB] Get:7 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release [6,051 B] Get:8 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current Release.gpg [490 B] Get:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] Get:10 http://downloads.linux.hpe.com/SDR/downloads/MCP/ubuntu bionic/current/non-free amd64 Packages [1,971 B] Fetched 352 kB in 1s (288 kB/s) Reading package lists... Done root@kb2018:/etc/apt# apt-get install ssacli ssaducli ... ``` Once the issues with his signature were resolved (above) I was able to instal the ssacli. [#fn2 (2)] #### Seeing the drives Use the ssacli to show the unassigned drives after inserting fresh disks. ```sh root@kb2018:/etc/apt# ssacli Smart Storage Administrator CLI 3.30.13.0 Detecting Controllers...Done. Type "help" for a list of supported commands. Type "exit" to close the console. => set target controller slot=0 "controller slot=0" => pd all show Smart Array P410i in Slot 0 (Embedded) Array A physicaldrive 2C:1:1 (port 2C:box 1:bay 1, SAS HDD, 146 GB, OK) physicaldrive 2C:1:2 (port 2C:box 1:bay 2, SAS HDD, 146 GB, OK) Array B physicaldrive 2C:1:3 (port 2C:box 1:bay 3, SATA SSD, 240 GB, OK) Array C physicaldrive 2C:1:4 (port 2C:box 1:bay 4, SATA SSD, 240 GB, OK) Array D physicaldrive 3C:1:5 (port 3C:box 1:bay 5, SAS HDD, 300 GB, OK) Array E physicaldrive 3C:1:6 (port 3C:box 1:bay 6, SAS HDD, 300 GB, OK) Unassigned physicaldrive 3C:1:7 (port 3C:box 1:bay 7, SAS HDD, 146 GB, OK) physicaldrive 3C:1:8 (port 3C:box 1:bay 8, SAS HDD, 146 GB, OK) ``` #### letting the OS see the drives Once we know what drives are available we can create logical drives which will be presented to the os _(assuming the same set target command above)_ ```sh => set target controller slot=0 ... => create type=ld drives=3C:1:7 size=max raid=0 => create type=ld drives=3C:1:8 size=max raid=0 quit ``` #### Preparing the drives for removal Before removing a drive you should delete the logical disk that it is associated with. ```sh => set target controller slot=0 ... => Array G delete ``` #### Increasing write performance once we get a ups we should be able to use the controllers write cache safely. ```sh => controller slot=0 modify drivewritecache=enable Warning: Without the proper safety precautions, use of write cache on physical drives could cause data loss in the event of power failure. To ensure data is properly protected, use redundant power supplies and Uninterruptible Power Supplies. Also, if you have multiple storage enclosures, all data should be mirrored across them. Use of this feature is not recommended unless these precautions are followed. Continue? (y/n) n => ``` ### footnotes [=#fn1 1]) This was discovered after digging around for the perccli raid utilities provided by dell (officially supported only on commercial RPM based systems but installable using alien) [=#fn2 2]) The biggest pain in the ass other than the weirdness with the public signature was that HP fucking rebranded the hpssacli to ssacli. Most of the good web info and hp docs still reference the old utility name (nothing else changed). ### references * http://h10032.www1.hp.com/ctg/Manual/c02289065.pdf (2010) * https://amk1.wordpress.com/2013/11/22/zfs-with-hp-smart-array-p410i/ * https://content.etilize.com/User-Manual/1033728289.pdf * http://www.sysadminshare.com/2012/05/hpacucli-commands-referrence.html * https://wiki.debian.org/LinuxRaidForAdmins * https://www.golinuxhub.com/2017/05/hot-swapping-broken-hdd-with-software.html * https://kallesplayground.wordpress.com/useful-stuff/hp-smart-array-cli-commands-under-esxi/ * http://downloads.linux.hpe.com/SDR/project/mcp/ * https://wiki.debian.org/HP/ProLiant#HP_Repository * https://binaryimpulse.com/2013/09/hp-array-configuration-utility-command-cheat-sheet/ * https://bibszone.wordpress.com/2016/02/11/hp-smart-array-cli-commands/ * https://h50146.www5.hpe.com/products/software/oe/linux/mainstream/support/doc/general/mgmt/ssa_cli/files/v240_130/hpssacli-2.40-13.0_help.txt * https://unixlab.weebly.com/raid-array.html * https://hardforum.com/threads/hp-dl380p-gen8-p420i-controller-hbamode.1852528/ ### addendum (output from ssacli show detailed config) =>ctrl all show config detail Smart Array P410i in Slot 0 (Embedded) Bus Interface: PCI Slot: 0 Serial Number: 5001438013631A40 Cache Serial Number: PBCDH0CRH0V0L0 Controller Status: OK Hardware Revision: C Firmware Version: 6.64-0 Rebuild Priority: Medium Expand Priority: Medium Surface Scan Delay: 15 secs Surface Scan Mode: Idle Parallel Surface Scan Supported: No Queue Depth: Automatic Monitor and Performance Delay: 60 min Elevator Sort: Enabled Degraded Performance Optimization: Disabled Wait for Cache Room: Disabled Surface Analysis Inconsistency Notification: Disabled Post Prompt Timeout: 0 secs Cache Board Present: True Cache Status: OK Cache Ratio: 25% Read / 75% Write Drive Write Cache: Disabled Total Cache Size: 0.5 Total Cache Memory Available: 0.4 No-Battery Write Cache: Disabled Cache Backup Power Source: Capacitors Battery/Capacitor Count: 1 Battery/Capacitor Status: OK SATA NCQ Supported: True Number of Ports: 2 Internal only Encryption: Not Set Driver Name: hpsa Driver Version: 3.4.20 Driver Supports SSD Smart Path: True PCI Address (Domain:Bus:Device.Function): 0000:05:00.0 Port Max Phy Rate Limiting Supported: False Host Serial Number: USE135N52V Sanitize Erase Supported: False Primary Boot Volume: None Secondary Boot Volume: None HP SAS Expander Card at Port 2C, Box 1, OK Power Supply Status: Not Redundant Vendor ID: HP Serial Number: RF15BP2689 Firmware Version: 2.10 Drive Bays: 24 Port: 2C Box: 1 Location: Internal Expander 250 Device Number: 250 Firmware Version: 2.10 WWID: 5001438014526C66 Box: 1 Vendor ID: HP HP SAS Expander Card SEP 248 Device Number: 248 Firmware Version: 2.10 Hardware Revision: Rev C WWID: 5001438014526C65 Box: 2 Vendor ID: HP Model: HP SAS EXP Card Physical Drives physicaldrive 2C:1:4 (port 2C:box 1:bay 4, SATA SSD, 240 GB, OK) physicaldrive 2C:1:3 (port 2C:box 1:bay 3, SATA SSD, 240 GB, OK) physicaldrive 2C:1:2 (port 2C:box 1:bay 2, SAS HDD, 146 GB, OK) physicaldrive 2C:1:1 (port 2C:box 1:bay 1, SAS HDD, 146 GB, OK) physicaldrive 3C:1:6 (port 3C:box 1:bay 6, SAS HDD, 300 GB, OK) physicaldrive 3C:1:5 (port 3C:box 1:bay 5, SAS HDD, 300 GB, OK) HP SAS Expander Card at Port 4C, Box 2, OK Power Supply Status: Not Redundant Vendor ID: HP Serial Number: RF15BP2689 Firmware Version: 2.10 Drive Bays: 24 Port: 4C Box: 2 Location: Internal Expander 250 Device Number: 250 Firmware Version: 2.10 WWID: 5001438014526C66 Box: 1 Vendor ID: HP HP SAS Expander Card SEP 248 Device Number: 248 Firmware Version: 2.10 Hardware Revision: Rev C WWID: 5001438014526C65 Box: 2 Vendor ID: HP Model: HP SAS EXP Card Physical Drives None attached Port Name: 1I Port ID: 0 Port Connection Number: 0 SAS Address: 5001438013631A40 Port Location: Internal Port Name: 2I Port ID: 1 Port Connection Number: 1 SAS Address: 5001438013631A44 Port Location: Internal Array: A Interface Type: SAS Unused Space: 6 MB (0.00%) Used Space: 273.40 GB (100.00%) Status: OK Array Type: Data Smart Path: disable Logical Drive: 1 Size: 136.70 GB Fault Tolerance: 1 Heads: 255 Sectors Per Track: 32 Cylinders: 35132 Strip Size: 256 KB Full Stripe Size: 256 KB Status: OK Unrecoverable Media Errors: None Caching: Enabled Unique Identifier: 600508B1001CAA24339C082CBF1B0912 Disk Name: /dev/sda Mount Points: / 80.0 GB Partition Number 2 OS Status: LOCKED Logical Drive Label: A0E0B9A75001438013631A40256F Mirror Group 1: physicaldrive 2C:1:2 (port 2C:box 1:bay 2, SAS HDD, 146 GB, OK) Mirror Group 2: physicaldrive 2C:1:1 (port 2C:box 1:bay 1, SAS HDD, 146 GB, OK) Drive Type: Data LD Acceleration Method: Controller Cache physicaldrive 2C:1:1 Port: 2C Box: 1 Bay: 1 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 146 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 15000 Firmware Revision: HPDD Serial Number: PLWGTWSE WWID: 5000CCA00B53489D Model: HP EH0146FARWD Current Temperature (C): 35 Maximum Temperature (C): 42 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown Sanitize Erase Supported: False Shingled Magnetic Recording Support: None physicaldrive 2C:1:2 Port: 2C Box: 1 Bay: 2 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 146 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 15000 Firmware Revision: HPDD Serial Number: PLWP0XNE WWID: 5000CCA00B5E9B11 Model: HP EH0146FARWD Current Temperature (C): 34 Maximum Temperature (C): 47 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown Sanitize Erase Supported: False Shingled Magnetic Recording Support: None Array: B Interface Type: Solid State SATA Unused Space: 2 MB (0.00%) Used Space: 223.54 GB (100.00%) Status: OK Array Type: Data Smart Path: disable Logical Drive: 2 Size: 223.54 GB Fault Tolerance: 0 Heads: 255 Sectors Per Track: 32 Cylinders: 57450 Strip Size: 256 KB Full Stripe Size: 256 KB Status: OK Caching: Enabled Unique Identifier: 600508B1001CC841DD71B0E330404FF4 Disk Name: /dev/sdb Mount Points: None Logical Drive Label: ABABB8965001438013631A40D1E0 Drive Type: Data LD Acceleration Method: Controller Cache physicaldrive 2C:1:3 Port: 2C Box: 1 Bay: 3 Status: OK Drive Type: Data Drive Interface Type: Solid State SATA Size: 240 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Firmware Revision: Q0410A Serial Number: AB20180827A0101371 WWID: 5001438014526C41 Model: ATA TEAML5Lite3D240G SATA NCQ Capable: True SATA NCQ Enabled: True SSD Smart Trip Wearout: Not Supported PHY Count: 1 PHY Transfer Rate: 3.0Gbps Sanitize Erase Supported: False Shingled Magnetic Recording Support: None Array: C Interface Type: Solid State SATA Unused Space: 2 MB (0.00%) Used Space: 223.54 GB (100.00%) Status: OK Array Type: Data Smart Path: disable Logical Drive: 3 Size: 223.54 GB Fault Tolerance: 0 Heads: 255 Sectors Per Track: 32 Cylinders: 57450 Strip Size: 256 KB Full Stripe Size: 256 KB Status: OK Caching: Enabled Unique Identifier: 600508B1001CD1056D9358D036DE54EB Disk Name: /dev/sdc Mount Points: None Logical Drive Label: ABAB89005001438013631A4045F6 Drive Type: Data LD Acceleration Method: Controller Cache physicaldrive 2C:1:4 Port: 2C Box: 1 Bay: 4 Status: OK Drive Type: Data Drive Interface Type: Solid State SATA Size: 240 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Firmware Revision: Q0410A Serial Number: AB20180827A0100293 WWID: 5001438014526C40 Model: ATA TEAML5Lite3D240G SATA NCQ Capable: True SATA NCQ Enabled: True SSD Smart Trip Wearout: Not Supported PHY Count: 1 PHY Transfer Rate: 3.0Gbps Sanitize Erase Supported: False Shingled Magnetic Recording Support: None Array: D Interface Type: SAS Unused Space: 0 MB (0.00%) Used Space: 279.37 GB (100.00%) Status: OK Array Type: Data Smart Path: disable Logical Drive: 4 Size: 279.37 GB Fault Tolerance: 0 Heads: 255 Sectors Per Track: 32 Cylinders: 65535 Strip Size: 256 KB Full Stripe Size: 256 KB Status: OK Caching: Enabled Unique Identifier: 600508B1001C868C26439B55D426224F Disk Name: /dev/sdd Mount Points: None Logical Drive Label: ABAB99875001438013631A40A72E Drive Type: Data LD Acceleration Method: Controller Cache physicaldrive 3C:1:5 Port: 3C Box: 1 Bay: 5 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 300 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 10000 Firmware Revision: HPD6 (FW update is recommended to minimum version: HPD7) Serial Number: PQJ0EM4B WWID: 5000CCA025718881 Model: HP EG0300FBDBR Current Temperature (C): 31 Maximum Temperature (C): 44 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown Sanitize Erase Supported: False Shingled Magnetic Recording Support: None Array: E Interface Type: SAS Unused Space: 0 MB (0.00%) Used Space: 279.37 GB (100.00%) Status: OK Array Type: Data Smart Path: disable Logical Drive: 5 Size: 279.37 GB Fault Tolerance: 0 Heads: 255 Sectors Per Track: 32 Cylinders: 65535 Strip Size: 256 KB Full Stripe Size: 256 KB Status: OK Caching: Enabled Unique Identifier: 600508B1001C380646CF15536E61E692 Disk Name: /dev/sde Mount Points: None Logical Drive Label: ABABE9D05001438013631A4088C8 Drive Type: Data LD Acceleration Method: Controller Cache physicaldrive 3C:1:6 Port: 3C Box: 1 Bay: 6 Status: OK Drive Type: Data Drive Interface Type: SAS Size: 300 GB Drive exposed to OS: False Logical/Physical Block Size: 512/512 Rotational Speed: 10000 Firmware Revision: HPD6 (FW update is recommended to minimum version: HPD7) Serial Number: PMVJ07DB WWID: 5000CCA0211D1B55 Model: HP EG0300FBDBR Current Temperature (C): 31 Maximum Temperature (C): 57 PHY Count: 2 PHY Transfer Rate: 6.0Gbps, Unknown Sanitize Erase Supported: False Shingled Magnetic Recording Support: None Expander 250 Device Number: 250 Firmware Version: 2.10 WWID: 5001438014526C66 Box: 1 Vendor ID: HP Expander 250 Device Number: 250 Firmware Version: 2.10 WWID: 5001438014526C66 Box: 1 Vendor ID: HP HP SAS Expander Card SEP 248 Device Number: 248 Firmware Version: 2.10 Hardware Revision: Rev C WWID: 5001438014526C65 Box: 2 Vendor ID: HP Model: HP SAS EXP Card HP SAS Expander Card SEP 248 Device Number: 248 Firmware Version: 2.10 Hardware Revision: Rev C WWID: 5001438014526C65 Box: 2 Vendor ID: HP Model: HP SAS EXP Card SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 249 Device Number: 249 Firmware Version: RevC WWID: 5001438013631A4F Vendor ID: PMCSIERA Model: SRC 8x6G