During the initial Exadata configuration at the step of creating cell disks we obtained the message "CELL-04532: Error encountered while creating cell disk on LUN":
INFO: Running /usr/local/bin/dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e create celldisk all " to disks...
ERROR: Ran /usr/local/bin/dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e create celldisk all " and it 1 instead of 0 and output is
...
ed02celadm01: CellDisk FD_00_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_01_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_02_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_03_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_04_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_05_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_06_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_07_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_08_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_09_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_10_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_11_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_12_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_13_ed02celadm01 successfully created
ed02celadm01:
ed02celadm01: CELL-04532: Error encountered while creating cell disk on LUN: 1_1 4_0 .
As you can see above there were created only 14 flash disks, not 16.
And disks 1_1 and 4_0 weren't created.
We connected to the storage cell and said list lun detail
CellCLI> list lun detail
name: 1_0
cellDisk: FD_00_ed02celadm01
deviceName: /dev/sdv
diskType: FlashDisk
id: 1_0
isSystemLun: FALSE
lunAutoCreate: FALSE
lunSize: 93.13225793838501G
physicalDrives: FLASH_1_0
status: normal
name: 1_1
cellDisk:
deviceName: /dev/sdw
diskType: FlashDisk
id: 1_1
isSystemLun: FALSE
lunAutoCreate: FALSE
lunSize: 93.13225793838501G
physicalDrives: FLASH_1_1
status: normal
As you can see - the LUN 1_1 has status "normal".
4_0 has the normal status too :).
And all good disks have the normal status too.
Then we run the disk reading:
[root@ed02celadm01 ~]# hdparm -tT /dev/sdv
/dev/sdv:
Timing cached reads: 25848 MB in 2.00 seconds = 12951.07 MB/sec
Timing buffered disk reads: 630 MB in 3.01 seconds = 209.62 MB/sec
[root@ed02celadm01 ~]# hdparm -tT /dev/sdw
/dev/sdw:
Timing cached reads: 26328 MB in 2.00 seconds = 13192.06 MB/sec
Timing buffered disk reads: 632 MB in 3.01 seconds = 210.04 MB/sec
As you can see, the bad and good disks are well both.
The "cellcli> CALIBRATE FORCE" showed the no errors too ...
Then we decided to create cell disks manually:
CellCLI> create celldisk FD_14_ed02celadm01 lun=1_1
CELL-04527: Cannot complete the creation of cell disk FD_14_ed02celadm01. Received error: CELL-04521: The LUN 1_1 has a valid celldisk (which is not imported).
Cell disks are not created: FD_14_ed02celadm01
CellCLI> create celldisk FD_14_ed02celadm01 lun=4_0
CELL-04527: Cannot complete the creation of cell disk FD_14_ed02celadm01. Received error: CELL-04521: The LUN 4_0 has a valid celldisk (which is not imported).
Cell disks are not created: FD_14_ed02celadm01
The solution is to run dd on the storage cell and wipe disk header:
dd if=/dev/zero of=/dev/sdw bs=1M count=20
INFO: Running /usr/local/bin/dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e create celldisk all " to disks...
ERROR: Ran /usr/local/bin/dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e create celldisk all " and it 1 instead of 0 and output is
...
ed02celadm01: CellDisk FD_00_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_01_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_02_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_03_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_04_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_05_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_06_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_07_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_08_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_09_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_10_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_11_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_12_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_13_ed02celadm01 successfully created
ed02celadm01:
ed02celadm01: CELL-04532: Error encountered while creating cell disk on LUN: 1_1 4_0 .
As you can see above there were created only 14 flash disks, not 16.
And disks 1_1 and 4_0 weren't created.
We connected to the storage cell and said list lun detail
CellCLI> list lun detail
name: 1_0
cellDisk: FD_00_ed02celadm01
deviceName: /dev/sdv
diskType: FlashDisk
id: 1_0
isSystemLun: FALSE
lunAutoCreate: FALSE
lunSize: 93.13225793838501G
physicalDrives: FLASH_1_0
status: normal
name: 1_1
cellDisk:
deviceName: /dev/sdw
diskType: FlashDisk
id: 1_1
isSystemLun: FALSE
lunAutoCreate: FALSE
lunSize: 93.13225793838501G
physicalDrives: FLASH_1_1
status: normal
As you can see - the LUN 1_1 has status "normal".
4_0 has the normal status too :).
And all good disks have the normal status too.
Then we run the disk reading:
[root@ed02celadm01 ~]# hdparm -tT /dev/sdv
/dev/sdv:
Timing cached reads: 25848 MB in 2.00 seconds = 12951.07 MB/sec
Timing buffered disk reads: 630 MB in 3.01 seconds = 209.62 MB/sec
[root@ed02celadm01 ~]# hdparm -tT /dev/sdw
/dev/sdw:
Timing cached reads: 26328 MB in 2.00 seconds = 13192.06 MB/sec
Timing buffered disk reads: 632 MB in 3.01 seconds = 210.04 MB/sec
As you can see, the bad and good disks are well both.
The "cellcli> CALIBRATE FORCE" showed the no errors too ...
Then we decided to create cell disks manually:
CellCLI> create celldisk FD_14_ed02celadm01 lun=1_1
CELL-04527: Cannot complete the creation of cell disk FD_14_ed02celadm01. Received error: CELL-04521: The LUN 1_1 has a valid celldisk (which is not imported).
Cell disks are not created: FD_14_ed02celadm01
CellCLI> create celldisk FD_14_ed02celadm01 lun=4_0
CELL-04527: Cannot complete the creation of cell disk FD_14_ed02celadm01. Received error: CELL-04521: The LUN 4_0 has a valid celldisk (which is not imported).
Cell disks are not created: FD_14_ed02celadm01
The solution is to run dd on the storage cell and wipe disk header:
dd if=/dev/zero of=/dev/sdw bs=1M count=20