Жесткий диск в программном RAID 5 исчез

У меня 6 жестких дисков в моем ящике, 2x256Gb ATA OCZ-AGILITY4 SSD.

Есть также 4x3.0Tb ATA ST3000DM001-9YN166, каждый из которых имеет один раздел, эти 4 раздела - RAID5 striped (md128). Все это ext4, отформатированный на сервере ubuntu 12.04. sda и sdb являются SSD, тогда как sdc, sdd, sde и sdf являются жесткими дисками.

Случайно (насколько я могу судить) md128 стал доступен только для чтения. Журнал sys выглядит так:

Aug 23 16:25:24 crick kernel: [617040.416257] ata4.00: exception Emask 0x0 SAct 0x1f SErr 0x0 action 0x0
Aug 23 16:25:24 crick kernel: [617040.416260] ata4.00: irq_stat 0x40000008
Aug 23 16:25:24 crick kernel: [617040.416262] ata4.00: failed command: READ FPDMA QUEUED
Aug 23 16:25:24 crick kernel: [617040.416265] ata4.00: cmd 60/08:08:00:af:cc/00:00:9c:00:00/40 tag 1 ncq 4096 in
Aug 23 16:25:24 crick kernel: [617040.416265]          res 41/40:08:00:af:cc/00:00:9c:00:00/00 Emask 0x409 (media error) <F>
Aug 23 16:25:24 crick kernel: [617040.416266] ata4.00: status: { DRDY ERR }
Aug 23 16:25:24 crick kernel: [617040.416267] ata4.00: error: { UNC }
Aug 23 16:25:24 crick kernel: [617040.417510] ata4.00: configured for UDMA/133
Aug 23 16:25:24 crick kernel: [617040.417527] sd 3:0:0:0: [sdd] Unhandled sense code
Aug 23 16:25:24 crick kernel: [617040.417528] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417529] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 23 16:25:24 crick kernel: [617040.417530] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417531] Sense Key : Medium Error [current] [descriptor]
Aug 23 16:25:24 crick kernel: [617040.417533] Descriptor sense data with sense descriptors (in hex):
Aug 23 16:25:24 crick kernel: [617040.417534]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
Aug 23 16:25:24 crick kernel: [617040.417538]         9c cc af 00 
Aug 23 16:25:24 crick kernel: [617040.417540] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417541] Add. Sense: Unrecovered read error - auto reallocate failed
Aug 23 16:25:24 crick kernel: [617040.417542] sd 3:0:0:0: [sdd] CDB: 
Aug 23 16:25:24 crick kernel: [617040.417543] Read(10): 28 00 9c cc af 00 00 00 08 00
Aug 23 16:25:24 crick kernel: [617040.417547] end_request: I/O error, dev sdd, sector 2630659840
Aug 23 16:25:24 crick kernel: [617040.417550] md/raid:md128: read error not correctable (sector 2630657792 on sdd1).
Aug 23 16:25:24 crick kernel: [617040.417552] md/raid:md128: Disk failure on sdd1, disabling device.
Aug 23 16:25:24 crick kernel: [617040.417552] md/raid:md128: Operation continuing on 2 devices.
Aug 23 16:25:24 crick kernel: [617040.417563] ata4: EH complete
Aug 23 16:25:25 crick kernel: [617040.455605] RAID conf printout:
Aug 23 16:25:25 crick kernel: [617040.455609]  --- level:5 rd:4 wd:2
Aug 23 16:25:25 crick kernel: [617040.455610]  disk 0, o:1, dev:sdc1
Aug 23 16:25:25 crick kernel: [617040.455611]  disk 1, o:0, dev:sdd1
Aug 23 16:25:25 crick kernel: [617040.455612]  disk 2, o:1, dev:sde1
Aug 23 16:25:25 crick kernel: [617040.489941] RAID conf printout:
Aug 23 16:25:25 crick kernel: [617040.489945]  --- level:5 rd:4 wd:2
Aug 23 16:25:25 crick kernel: [617040.489947]  disk 0, o:1, dev:sdc1
Aug 23 16:25:25 crick kernel: [617040.489948]  disk 2, o:1, dev:sde1

A slew of:
Aug 23 16:25:25 crick kernel: [617040.539926] Buffer I/O error on device md128, logical block 986401023
with different block addresses then

Aug 23 16:25:25 crick kernel: [617040.539929] EXT4-fs warning (device md128): ext4_end_bio:248: I/O error writing to inode 42993727 (offset 11551637504 size 524288 starting block 986400896)
Aug 23 16:25:25 crick kernel: [617040.541690] JBD2: Detected IO errors while flushing file data on md128-8
Aug 23 16:25:25 crick kernel: [617040.541707] Aborting journal on device md128-8.
Aug 23 16:25:25 crick kernel: [617040.541720] EXT4-fs error (device md128) in ext4_free_blocks:4702: Journal has aborted
Aug 23 16:25:25 crick kernel: [617040.541727] Buffer I/O error on device md128, logical block 1098416128
Aug 23 16:25:25 crick kernel: [617040.541729] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.541734] Buffer I/O error on device md128, logical block 0
Aug 23 16:25:25 crick kernel: [617040.541736] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.541740] JBD2: Error -5 detected when updating journal superblock for md128-8.
Aug 23 16:25:25 crick kernel: [617040.541743] EXT4-fs (md128): delayed block allocation failed for inode 49152114 at logical offset 994912 with max blocks 2048 with error -30
Aug 23 16:25:25 crick kernel: [617040.541745] EXT4-fs (md128): This should not happen!! Data will be lost
Aug 23 16:25:25 crick kernel: [617040.541745] 
Aug 23 16:25:25 crick kernel: [617040.541806] EXT4-fs (md128): previous I/O error to superblock detected
Aug 23 16:25:25 crick kernel: [617040.542518] EXT4-fs error (device md128) in ext4_da_writepages:2390: Journal has aborted
Aug 23 16:25:25 crick kernel: [617040.542526] JBD2: Detected IO errors while flushing file data on md128-8
Aug 23 16:25:25 crick kernel: [617040.542529] Buffer I/O error on device md128, logical block 0
Aug 23 16:25:25 crick kernel: [617040.542531] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.542798] EXT4-fs error (device md128): ext4_journal_start_sb:371: Detected aborted journal
Aug 23 16:25:25 crick kernel: [617040.542810] EXT4-fs (md128): Remounting filesystem read-only
Aug 23 16:25:25 crick kernel: [617040.542815] EXT4-fs (md128): previous I/O error to superblock detected
Aug 23 16:25:25 crick kernel: [617040.542835] EXT4-fs (md128): I/O error while writing superblock
Aug 23 16:25:25 crick kernel: [617040.542838] EXT4-fs (md128): ext4_da_writepages: jbd2_start: 15984 pages, ino 49152114; err -30
Aug 23 16:25:25 crick kernel: [617040.544887] EXT4-fs error (device md128): ext4_journal_start_sb:371: Detected aborted journal

Когда я пользуюсь дисковой утилитой или gparted sdf ушел, он даже не обнаруживается. Поскольку md128 деградирует. Первое, что я сделал, это размонтировать его с помощью дисковой утилиты, после чего я попытался посмотреть, может ли что-нибудь еще найти:

fdisk -l

Disk /dev/sda: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a88fe

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1       406368256   500117503    46874624   fd  Linux raid autodetect
/dev/sda2            2046   406368255   203183105    5  Extended
/dev/sda5            2048   406368255   203183104   fd  Linux raid autodetect

Partition table entries are not in disk order

Disk /dev/md0: 96.0 GB, 95998181376 bytes
2 heads, 4 sectors/track, 23437056 cylinders, total 187496448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 207.9 GB, 207925149696 bytes
2 heads, 4 sectors/track, 50762976 cylinders, total 406103808 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/sdb: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b0740

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *   406368256   500117503    46874624   fd  Linux raid autodetect
/dev/sdb2            2046   406368255   203183105    5  Extended
/dev/sdb5            2048   406368255   203183104   fd  Linux raid autodetect

Partition table entries are not in disk order

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

Disk /dev/md128: 9001.4 GB, 9001370124288 bytes
2 heads, 4 sectors/track, -2097367168 cylinders, total 17580801024 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1572864 bytes
Disk identifier: 0x00000000

Disk /dev/md128 doesn't contain a valid partition table

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

cat / etc / fstab

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md127 during installation
UUID=b63b7341-0b85-40ed-a67b-acfe4f65f563 /               ext4    errors=remount-ro 0       1
# /genome was on /dev/md128 during installation
UUID=bd6b54be-12ca-479d-879e-8d788fa9d039 /genome         ext4    defaults        0       2
# swap was on /dev/md126 during installation
UUID=0470550a-6e92-485d-ad41-665c3f313287 none            swap    sw              0       0

~ # cat / proc / mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md128 : active raid5 sde1[2] sdc1[0]
      8790400512 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2] [U_U_]

md1 : active raid1 sdb5[1] sda5[0]
      203051904 blocks super 1.2 [2/2] [UU]

md0 : active raid0 sdb1[1] sda1[0]
      93748224 blocks super 1.2 512k chunks

unused devices: <none>

~ # mdadm --detail --scan

ARRAY /dev/md/0 metadata=1.2 name=crick:0 UUID=d0d08eab:a7e54021:25973acb:10dd5fba
ARRAY /dev/md/1 metadata=1.2 name=crick:1 UUID=e2774945:5b3ee3eb:2ad9390f:35153b82
ARRAY /dev/md/128 metadata=1.2 name=crick:128 UUID=345cb755:0ae1c919:d98a45ca:1baf3364

Вот где мое понимание linux не самый лучший. Из того, что я могу сказать, суперблок массива видит UUID для sdf, на котором он был построен, но (по какой-либо причине) sdf не обнаруживается. Это означает, что RAID-массив поврежден, поэтому ubuntu установил файловую систему в RO в качестве меры предосторожности. Я все еще могу открывать, редактировать и создавать файлы без проблем, ОС по-прежнему работает отлично, я просто не могу получить доступ к данным на md128.

Примерно неделю назад произошла одна и та же проблема (случайный переход к RO), только в это время она ударила md1 ( SSD). Когда это произошло, суперблок был удален на md128. Мне повезло при переустановке ОС и пересоздании RAID с использованием существующих разделов (IE не форматировал разделы), и удалось восстановить 99,9% данных (файлы, к которым обращались, когда это произошло, были повреждены).

EDIT: Извините за OP, я новичок в форуме и не читал инструкции по форматированию, теперь я исправил его