CELL-04532: Error encountered while creating cell disk on LUN

During the initial Exadata configuration at the step of creating cell disks we obtained the message "CELL-04532: Error encountered while creating cell disk on LUN":

INFO: Running /usr/local/bin/dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e create celldisk all " to disks...
ERROR: Ran /usr/local/bin/dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e create celldisk all " and it 1 instead of 0 and output is
 
...
ed02celadm01: CellDisk FD_00_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_01_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_02_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_03_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_04_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_05_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_06_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_07_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_08_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_09_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_10_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_11_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_12_ed02celadm01 successfully created
ed02celadm01: CellDisk FD_13_ed02celadm01 successfully created
ed02celadm01:
ed02celadm01: CELL-04532: Error encountered while creating cell disk on LUN: 1_1 4_0 .

As you can see above there were created only 14 flash disks, not 16.
And disks 1_1 and 4_0 weren't created.

We connected to the storage cell and said list lun detail

CellCLI> list lun detail

         name:                   1_0
         cellDisk:               FD_00_ed02celadm01
         deviceName:             /dev/sdv
         diskType:               FlashDisk
         id:                     1_0
         isSystemLun:            FALSE
         lunAutoCreate:          FALSE
         lunSize:                93.13225793838501G
         physicalDrives:         FLASH_1_0
         status:                 normal

         name:                   1_1
         cellDisk:
         deviceName:             /dev/sdw
         diskType:               FlashDisk
         id:                     1_1
         isSystemLun:            FALSE
         lunAutoCreate:          FALSE
         lunSize:                93.13225793838501G
         physicalDrives:         FLASH_1_1
         status:                 normal

As you can see - the LUN 1_1 has status "normal".
4_0 has the normal status too :).
And all good disks have the normal status too.


Then we run the disk reading:

[root@ed02celadm01 ~]# hdparm -tT /dev/sdv

/dev/sdv:
 Timing cached reads:   25848 MB in  2.00 seconds = 12951.07 MB/sec
 Timing buffered disk reads:  630 MB in  3.01 seconds = 209.62 MB/sec
 
[root@ed02celadm01 ~]# hdparm -tT /dev/sdw
/dev/sdw:
 Timing cached reads:   26328 MB in  2.00 seconds = 13192.06 MB/sec
 Timing buffered disk reads:  632 MB in  3.01 seconds = 210.04 MB/sec


As you can see, the bad and good disks are well both.

The "cellcli> CALIBRATE FORCE" showed the no errors too ...


Then we decided to create cell disks manually:

CellCLI> create celldisk FD_14_ed02celadm01 lun=1_1

CELL-04527: Cannot complete the creation of cell disk FD_14_ed02celadm01. Received error: CELL-04521: The LUN 1_1 has a valid celldisk (which is not imported).
Cell disks are not created: FD_14_ed02celadm01

CellCLI> create celldisk FD_14_ed02celadm01 lun=4_0

CELL-04527: Cannot complete the creation of cell disk FD_14_ed02celadm01. Received error: CELL-04521: The LUN 4_0 has a valid celldisk (which is not imported).
Cell disks are not created: FD_14_ed02celadm01

The solution is to run dd on the storage cell and wipe disk header:


dd if=/dev/zero of=/dev/sdw bs=1M count=20

Command: ChgPassWordString grid welcome1 produced null output

Making new installation of 1/2 Exadata i've got a message:

[root@dm01dbadm01 linux]# ./install.sh -cf ./WorkDir/MegaYEKTTest.xml -s 3

20 Sep 13 14:19:45 [INFO ] Executing Create Users
20 Sep 13 14:19:45 [INFO ] Creating users...
20 Sep 13 14:19:45 [INFO ] Creating users in cluster cluster-clu1 ................................
20 Sep 13 14:20:23 [INFO ] Following errors were found while checking command output:
20 Sep 13 14:20:23 [INFO ] ERROR:
20 Sep 13 14:20:23 [INFO ] Command: ChgPassWordString grid welcome1 produced null output but executed successfully on dm01dbadm03
20 Sep 13 14:20:23 [INFO ] zipping log and WorkDir directories . . 
20 Sep 13 14:20:23 [INFO ] Please send /opt/oracle.SupportTools/linux/WorkDir/Diag-130920_142023.zip to Oracle if you require assistance...
20 Sep 13 14:20:23 [INFO ] OcmdException from node dm01dbadm01.mega.com return code = 2 output string: Error running command ChgPassWordString grid welcome1 on node dm01dbadm03
20 Sep 13 14:20:23 [INFO ] OcmdException from node dm01dbadm01.mega.com return code = 2 output string: Error running Create Users error message Error running oracle.onecommand.deploy.users.DeployUserUtils method createAllUsers






I logined to node 3 and 4 and tried to change password manually, and confirmed the error:

[root@dm01dbadm04 ~]# passwd
Changing password for user root.
... some silent seconds and .
passwd: Authentication token manipulation error

But in node 1 and node 2 passwd worked well.

I did
# strace passwd
and noticed all files passwd is open
open("/etc/pam.d/system-auth", O_READONLY) =...
They are
/etc/passwd, /etc/shadow, /etc/pam.d/passwd, /etc/pam.d/system-auth, /etc/pam.d/other

 Then I compared  these files at good and bad nodes.
Actually the difference was in some commented lines in /etc/pam.d/system-auth in bad node.  

 SOLUTION


I copied files from good node to bad nodes:
/etc/passwd, /etc/shadow, /etc/pam.d/passwd, /etc/pam.d/system-auth, /etc/pam.d/other


And the problem was gone :) !