BitPrison: How to repair raid array

This guide shows how to repair your MD RAID1 array on Ubuntu server. Not sure but I assume all linux work this way. So this solution must work on other distributions.

1. Identify the bad disk

Use smartctl to identify the bad disk(s):

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[2] sda1[0]
      960805696 blocks super 1.2 [2/2] [UU]

Now, it shows the array is up and running. Let check the disks:

smartctl --all /dev/sda
smartctl --all /dev/sdb

After it, I saw the /dev/sdb disk is failed.

2. Remove bad disk from array:

$ mdadm --manage /dev/md0 --fail /dev/sdb1
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[2](F) sda1[0]
      960805696 blocks super 1.2 [2/1] [U_]
$ mdadm --manage /dev/md0 --remove /dev/sdb1
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda1[0]
      960805696 blocks super 1.2 [2/1] [U_]

unused devices: <none>

3. Shutdown and replace the bad disk.

sudo shutdown -h now

4. Rebuild the RAID 1 array

Copy the partition table:

sfdisk -d /dev/sda | sfdisk /dev/sdb

Add new HDD to the MD0 RAID 1 Array:

mdadm --manage /dev/md0 --add /dev/sdb1

5. Waiting for rebuilding

Check the progress of rebuilding:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[2] sda1[0]
      960805696 blocks super 1.2 [2/1] [U_]
      [>....................]  recovery =  0.3% (3542720/960805696) finish=135.0min speed=118090K/sec

unused devices: <none>

Control the I/O resources of rebuilding process (Optional)

You can control the bandwith of RAID rebuilding process. First check the current speed that found in the following file:

$ cat /sys/block/md0/md/sync_speed_max
200000 (system)

After it you can change the bandwith. The speed is in Kibibytes per second. The system wide default value is in /proc/sys/dev/raid/speedlimitmax file.

If I like to change the max speed from 200000 to 30000 I need to write this value into the mentioned file:

$ echo 30000 > /sys/block/md1/md/sync_speed_max

You can control the minimal bandwith too. Just replace the max with min like sync_speed_min and speed_limit_min.

6. Our RAID1 array is up again

After few hours:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[2] sda1[0]
      960805696 blocks super 1.2 [2/2] [UU]

7. Should make sure about grub installed everywhere

First you should reboot once raid array is up.

And tell grub that we need a grub installation on the new hdd.

dpkg-reconfigure grub-pc

Here you can check disks in a list. Make sure about the new disk is checked.

Sources:

http://linux.die.net/man/4/md https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array