Check the arrays
$ cat /proc/mdstat
$ mdadm --detail /dev/md2A healthy RAID 1 shows [2/2] [UU]. [2/1] [U_] means one member is missing or failed.
Identify the failed drive
$ mdadm --detail /dev/md2 | grep -E 'faulty|removed'
$ nvme list # NVMe: model and serial number
$ smartctl -i /dev/sdb # SATA/SAS drivesOpen a ticket with the serial number of the failed drive (and of the healthy one, so we pull the right disk). Failed disks are replaced within 2 hours.
Rebuild after the swap
Copy the partition table from the healthy disk to the new one, give it new identifiers, then add its partitions back to each array. In this example nvme0n1 is healthy and nvme1n1 is new:
$ sgdisk -R /dev/nvme1n1 /dev/nvme0n1 # copy table TO the new disk
$ sgdisk -G /dev/nvme1n1 # randomise its GUIDs
$ mdadm /dev/md0 --add /dev/nvme1n1p2
$ mdadm /dev/md2 --add /dev/nvme1n1p3
$ watch -n 5 cat /proc/mdstatOrder matters: sgdisk -R copies to the first device from the second. Swapping them would wipe the healthy disk’s table.
Finally, reinstall the boot loader on the new disk (grub-install /dev/nvme1n1 on BIOS systems, or copy the EFI system partition on UEFI systems) so the server boots from either drive.
Get alerted next time
Set MAILADDR you@example.org in /etc/mdadm/mdadm.conf and make sure the mdmonitor service runs: mdadm emails you as soon as an array degrades.