|
|
skyBuilders HOWTO Construct, Test, and Maintain a RAID Fedora Core 4 Host
I. Install and configure the system
i. Install a new system on the host
- Run the regular Fedora Core installer from the CD/DVD
- Choose Disk Druid for partitioning.
- Delete all existing partitions. Disk Druid will suggest some LVM volumes; they may require sequenced deletion of partitions.
- On each hard drive, make RAID partitions for /boot (100MB), /swap (2xRAM), and / (remaining free space) on each of the available drives (actual mount points are selected when the partitions are merged in the next step). File System Type should be set to Software RAID for all partitions.
- Once the partitions are created, merge the three matching sets of partitions into arrays using the RAID button. For each array:
- enter the appropriate mount point (where applicable),
- select the appropriate file system (swap for /swap, ext3 for the others),
- select the appropriate RAID Device (md0 for /boot, md1 for /swap, md2 for /), and
- select RAID Level RAID1
For further information, Red Hat has a good guide page with screenshots of the Disk Druid interface and details of how to make proper selections for RAIDing.
- Add packages normally for the type of host being created and complete installation.
ii. Configure RAID to be bootable
- After full installation, the machine will reboot to the grub prompt (if it boots directly into linux, just log in and run the grub command to get to the grub prompt). Now the RAID boot partitions on each drive must be setup for grub. They are known to grub as hd0, hd1, etc. At the boot prompt, repeat the following commands for each drive:
grub> root (hd0,0)
grub> setup (hd0)
Once all of the drives have been setup, if the machine booted into grub, enter the command reboot to reboot the system (the machine will then reboot to a normal running state), otherwise enter the command quit to exit the grub console.
- Make a bootable CD image (the device parameter should be renamed to correspond with the new hostname; the last parameter must match the kernel version):
# mkbootdisk --device ./hostsa_bootdisk.iso --iso 2.6.11-1.1369_FC4smp
Move this image into an area where it will be backed up for emergency retrieval.
- View overall RAID status using mdstat:
# less /proc/mdstat
- Edit /boot/grub/grub.conf to list both drives for bootability, so that either one will be independently capable of booting:
- Backup /boot/grub/grub.conf
# cp /boot/grub/grub.conf /boot/grub/grub.conf.orig
- Open /boot/grub/grub.conf for editing in your favorite editor. For each extra hard drive, make a copy of each section starting with "title", replacing all refreferences to hd0 with hd# (like hd1 for the first extra drive, hd2 for the second, etc.), leaving intact the sections that refer to hd0. Once the edits are complete, save the file and close it.
- Verify that /boot/grub/device.map lists all drives in the array so grub will recognize both drives; if any sd device is missing, add an appropriate line to the file. Here's what should be in the file by default for two SATA drives:
# less /boot/grub/device.map
(fd0) /dev/fd0
(hd0) /dev/sda
(hd1) /dev/sdb
- Enable the RAID monitor to send an email upon failure (-m == --monitor, -f == --daemonise, outputs the pid of the monitor process to /var/run/mdadm):
# mdadm --monitor --scan --daemonise --mail=monitor@skybuilders.com > /var/run/mdadm/mdadm.pid
(To get the full advantage of this usage of mdadm edit the file, /etc/rc.local, to have the above command at the end of the file. The rc.local file is executed at the end of the operating system boot process, ensuring that the daemonized mdadm command is launched at boot time.)
- Disable graphical boot (especially important since current FC4 ISOs have video driver bugs that may cause the graphical boot to fail):
# emacs /etc/inittab
change the line:
id:5:initdefault:
to read:
id:3:initdefault:
- test reboot (under FC4, reboot should succeed without error, though make sure to skip past the Linux Setup Agent that FC distributions run on first boot):
# shutdown -r now
II. Test/Maintain the RAID
i. Failure
- If a drive fails, a notification email should be sent by the host. If such a notification has been received or if a spot check is in progress, RAID status should be checked (example output for a two-drive array of 80GB drives in optimal mode follows, your blocks may vary):
# less /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[1] sda2[0]
1020032 blocks [2/2] [UU]
md2 : active raid1 sdb3[1] sda3[0]
76999424 blocks [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]
unused devices: <none>
For each array, [2/2] [UU] is shown to indicate that of two drives in the array two drives are working and present. If a drive has failed, [2/1] will be shown and there will be an _ instead of one of the U's listed above for each array to show that the array is in degraded mode. On the line above that, each constituent partition is listed by device name.
- If this is a test of RAID failure and recovery, then degrade the three devices on one drive to simulate failure (this is NOT necessary in a real failure case) using the mdadm command:
# mdadm /dev/md0 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
# mdadm /dev/md1 -f /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md1
# mdadm /dev/md2 -f /dev/sdb3
mdadm: set /dev/sdb3 faulty in /dev/md2
- Check RAID status during failure. If there is a real or simulated drive failure, the status should look like this:
# less /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[2](F) sda2[0]
1020032 blocks [2/1] [U_]
md2 : active raid1 sdb3[2](F) sda3[0]
76999424 blocks [2/1] [U_]
md0 : active raid1 sdb1[2](F) sda1[0]
104320 blocks [2/1] [U_]
unused devices: <none>
The status now shows [2/1] [U_] for each array, showing that it currently consists of two partitions, one of which is degraded. On the line above that, an (F) is placed after the device name of the failed partition.
- Remove the devices from the arrays (this is necessary both for testing and for a real failure maintenance) (be sure to substitute the proper device names of the actual degraded partitions, here /dev/sdb1, 2, and 3):
# mdadm /dev/md0 -r /dev/sdb1
mdadm: hot removed /dev/sdb1
# mdadm /dev/md1 -r /dev/sdb2
mdadm: hot removed /dev/sdb2
# mdadm /dev/md2 -r /dev/sdb3
mdadm: hot removed /dev/sdb3
- Check RAID status:
# less /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda2[0]
1020032 blocks [2/1] [U_]
md2 : active raid1 sda3[0]
76999424 blocks [2/1] [U_]
md0 : active raid1 sda1[0]
104320 blocks [2/1] [U_]
unused devices: <none>
The status still shows [2/1] [U_] for each array, to show degraded mode. The line above that now lists only one device for each array, showing that the failed device has been removed.
ii. Reconstruction
- If this is a real failure case, shutdown the machine and replace the failed drive with one that matches exactly for size and architecture. Once the drive is in place, boot and login.
- Use fdisk to reformat the test-degraded or replacement drive (in the example, /dev/sdb), (for test, deleting all partitions and) recreating the partitions of the original exactly, including type and bootability (mount point has already been established by the RAID, so it is unnecessary here). This will wipe all data from the drive, making it a newly formatted drive. The fdisk console makes changes to a draft table; they will not be committed to the real partition table unless the w command is used to save/write the changes. If the fdisk console is exited (by the q command or control-c), all unsaved changes will be lost. However, the w command not only saves changes but also exits the console, so it should not be invoked until all changes are complete.
- For reference, examine the partitioning of a healthy drive (in the example /dev/sda) using the fdisk command to enter the fdisk console (explanations of command letters follow in square brackets; do not type these):
# fdisk /dev/sda
Command (m for help): p [print the partition table]
Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 13 104391 fd Linux raid autodetect
/dev/sdb2 14 140 1020127+ fd Linux raid autodetect
/dev/sdb3 141 9726 76999545 fd Linux raid autodetect
Command (m for help): q [quit without saving changes]
- Now repartition the degraded/replacement drive starting by deleting any existing partitions (in this example, the test partitions will be deleted; in a real failure case, check for any existing partitions using p and delete them using this example as a model):
# fdisk /dev/sdb
Command (m for help): d [delete a partition]
Partition number (1-4): 1
Command (m for help): d [delete a partition]
Partition number (1-4): 2
Command (m for help): d [delete a partition, autoselects remaining partition]
Selected partition 3
- Now add new partitions that match the original partitioning, using the partition table from the healthy drive as a guide:
Command (m for help): n [add a new partition]
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-9726, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-9726, default 9726): 13
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14-9726, default 14): 14
Last cylinder or +size or +sizeM or +sizeK (14-9726, default 9726): 140
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (141-9726, default 141):
Using default value 141
Last cylinder or +size or +sizeM or +sizeK (141-9726, default 9726):
Using default value 9726
- Change the type/id of each new partition to fd, which is the hex code for Linux raid auto:
Command (m for help): t [change a partition's system id]
Partition number (1-4): 1
Hex code (type L to list codes): fd [Linux raid auto]
Changed system type of partition 1 to fd (Linux raid autodetect)
Command (m for help): t [change a partition's system id]
Partition number (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)
Command (m for help): t [change a partition's system id]
Partition number (1-4): 3
Hex code (type L to list codes): fd
Changed system type of partition 3 to fd (Linux raid autodetect)
- Set the /boot partition bootable:
Command (m for help): a [toggle a bootable flag]
Partition number (1-4): 1
- Use 'p' to check the partition table and verify that it matches the table from the healthy drive.
- Save the changes made to the partition table and exit the fdisk console:
Command (m for help): w [write table to disk and exit]
- Add new partition devices to RAID arrays (this will automatically start the array reconstruction process on each one):
# mdadm /dev/md0 -a /dev/sdb1
# mdadm /dev/md1 -a /dev/sdb2
# mdadm /dev/md2 -a /dev/sdb3
- Periodically check RAID status to monitor reconstruction process:
# less /proc/mdstat
Reconstruction progress will be shown for each array. While reconstruction continues, an array will be displayed in degraded mode with all drives present but one degraded. Once reconstruction is complete, the array will be displayed in optimal mode. Reconstruction (and any test) should now be complete and successful.
III. Reference material
|
