"Found 3 configured voting files but 2 voting files are required" after upgrade to 12.2.0.1.180717

After upgrade Grid Infrastructure from 12.1 to 12.2 we obtained the message:

CRS-1705: Found 3 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc

The all steps of upgrade were successful. There were no errors on the screen and in the log files.
No database soft was touched, no OS software was touched, the only GI was upgraded.
But all 4 virtual machines upgraded from 12.1 to 12.2.0.1.180817 were in bad state: ClusterWare not started automatically. The Linux restart did not help.

 Previous upgrade of 5th VM few day ago was successful and we obtained the working configuration.

The CW alert log is:
[CSSDMONITOR(121305)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 121305
[CSSDAGENT(121338)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 121338
[OHASD(224984)]CRS-2878: Failed to restart resource 'ora.storage'
[ORAROOTAGENT(228831)]CRS-5021: Check of storage failed: details at "(:CLSN00117:)" in "/u01/app/oracle/diag/crs/mr01vm03/crs/trace/ohasd_orarootagent_root.trc"
[CSSDAGENT(123152)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 123152
[OCSSD(123165)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 123165
[OCSSD(123165)]CRS-1713: CSSD daemon is started in hub mode
[OCSSD(123165)]CRS-1705: Found 3 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc
[OCSSD(123165)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc
[OCSSD(123165)]CRS-1652: Starting clean up of CRSD resources.
[OCSSD(123165)]CRS-1653: The clean up of the CRSD resources failed.
[OCSSD(123165)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 123165 experienced fatal signal or exception code 6.
Errors in file /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc  (incident=729): CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/oracle/diag/crs/mr01vm03/crs/incident/incdir_729/ocssd_i729.trc

Very strange message: "Found 3 configured voting files but 2 voting files are required,"

SOLUTION:  move voting files to other ASM diskgroup.

The commands log:

[root@mr01vm03 ~]# cd /u01/app/12.2.0.1/grid/bin/
[root@mr01vm03 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.crf' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.diskmon' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.evmd' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'mr01vm03'
CRS-2677: Stop of 'ora.drivers.acfs' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.crf' on 'mr01vm03' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'mr01vm03'
CRS-2677: Stop of 'ora.gpnpd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.evmd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.cssdmonitor' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'mr01vm03' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mr01vm03' has completed
CRS-4133: Oracle High Availability Services has been stopped.
 
[root@mr01vm03 bin]# ./crsctl start crs -excl
 
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'mr01vm03'
CRS-2672: Attempting to start 'ora.mdnsd' on 'mr01vm03'
CRS-2676: Start of 'ora.mdnsd' on 'mr01vm03' succeeded
CRS-2676: Start of 'ora.evmd' on 'mr01vm03' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'mr01vm03'
CRS-2676: Start of 'ora.gpnpd' on 'mr01vm03' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'mr01vm03'
CRS-2672: Attempting to start 'ora.gipcd' on 'mr01vm03'
CRS-2676: Start of 'ora.cssdmonitor' on 'mr01vm03' succeeded
CRS-2676: Start of 'ora.gipcd' on 'mr01vm03' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'mr01vm03'
CRS-2672: Attempting to start 'ora.diskmon' on 'mr01vm03'
CRS-2676: Start of 'ora.diskmon' on 'mr01vm03' succeeded
CRS-2676: Start of 'ora.cssd' on 'mr01vm03' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'mr01vm03'
CRS-2672: Attempting to start 'ora.ctssd' on 'mr01vm03'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'mr01vm03'
CRS-2676: Start of 'ora.crf' on 'mr01vm03' succeeded
CRS-2676: Start of 'ora.ctssd' on 'mr01vm03' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'mr01vm03' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'mr01vm03'
CRS-2676: Start of 'ora.asm' on 'mr01vm03' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'mr01vm03'
CRS-2676: Start of 'ora.storage' on 'mr01vm03' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'mr01vm03'
CRS-2676: Start of 'ora.crsd' on 'mr01vm03' succeeded




CW 12.2 log:
[OHASD(126854)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 126854
[OHASD(126854)]CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
[OHASD(126854)]CRS-2112: The OLR service started on node mr01vm03.
[OHASD(126854)]CRS-1301: Oracle High Availability Service started on node mr01vm03.
[OHASD(126854)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
Create Relation ADR_CONTROL_AUX
Create Relation DFW_PURGE
[CSSDAGENT(126987)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 126987
[ORAROOTAGENT(126973)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 126973
[CSSDMONITOR(126992)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 126992
[ORAAGENT(126983)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 126983
[ORAAGENT(127057)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 127057
[EVMD(127078)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 127078
[MDNSD(127076)]CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 127076
[GPNPD(127143)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 127143
Create Relation DFW_PURGE_ITEM
[CSSDMONITOR(127195)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 127195
[GIPCD(127197)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 127197
[GPNPD(127143)]CRS-2328: GPNPD started on node mr01vm03.
[CSSDAGENT(127235)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 127235
[OCSSD(127253)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 127253
[OCSSD(127253)]CRS-1713: CSSD daemon is started in hub mode
[OCSSD(127253)]CRS-1707: Lease acquisition for node mr01vm03 number 1 completed
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.5;192.168.10.6/DATAC3_CD_02_mrceladm01; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.7;192.168.10.8/DATAC3_CD_02_mrceladm02; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.9;192.168.10.10/DATAC3_CD_02_mrceladm03; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1601: CSSD Reconfiguration complete. Active nodes are mr01vm03 .
[OCSSD(127253)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
[OSYSMOND(127434)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 127434
[OCTSSD(127440)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 127440
[OCTSSD(127440)]CRS-2403: The Cluster Time Synchronization Service on host mr01vm03 is in observer mode.
[OCTSSD(127440)]CRS-2407: The new Cluster Time Synchronization Service reference node is host mr01vm03.
[OCTSSD(127440)]CRS-2401: The Cluster Time Synchronization Service started on host mr01vm03.
[ORAAGENT(127057)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/u01/app/oracle/diag/crs/mr01vm03/crs/trace/ohasd_oraagent_oracle.trc"
[CRSD(127895)]CRS-8500: Oracle Clusterware CRSD process is starting with operating system process ID 127895
[CRSD(127895)]CRS-1012: The OCR service started on node mr01vm03.


[root@mr01vm03 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   5789f55a211a4fadbf851c10d7d2f56d (o/192.168.10.7;192.168.10.8/DATAC3_CD_02_mrceladm02) [DATAC3]
2. ONLINE   00a2d7ffaa294f77bf5e1941f5f3f9b3 (o/192.168.10.5;192.168.10.6/DATAC3_CD_02_mrceladm01) [DATAC3]
3. ONLINE   7179ec74f6474f96bfd0cbace76f4e6e (o/192.168.10.9;192.168.10.10/DATAC3_CD_02_mrceladm03) [DATAC3]
Located 3 voting disk(s).
 
[root@mr01vm03 bin]# ./crsctl replace votedisk +RECOC3
 
Successful addition of voting disk 2d8f371e02d54ffabf33e189ed5c42c7.
Successful addition of voting disk dbba5f46190a4f1dbf73753552e2e82b.
Successful addition of voting disk d6caa92e49234f9ebfe6fe749ea7dac9.
Successful deletion of voting disk 5789f55a211a4fadbf851c10d7d2f56d.
Successful deletion of voting disk 00a2d7ffaa294f77bf5e1941f5f3f9b3.
Successful deletion of voting disk 7179ec74f6474f96bfd0cbace76f4e6e.
Successfully replaced voting disk group with +RECOC3.
CRS-4266: Voting file(s) successfully replaced

CW 12.2. log:
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.7;192.168.10.8/RECOC3_CD_02_mrceladm02; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.5;192.168.10.6/RECOC3_CD_04_mrceladm01; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.9;192.168.10.10/RECOC3_CD_02_mrceladm03; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1604: CSSD voting file is offline: o/192.168.10.5;192.168.10.6/DATAC3_CD_02_mrceladm01; details at (:CSSNM00069:) in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1604: CSSD voting file is offline: o/192.168.10.7;192.168.10.8/DATAC3_CD_02_mrceladm02; details at (:CSSNM00069:) in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1604: CSSD voting file is offline: o/192.168.10.9;192.168.10.10/DATAC3_CD_02_mrceladm03; details at (:CSSNM00069:) in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1626: A Configuration change request completed successfully
[OCSSD(127253)]CRS-1601: CSSD Reconfiguration complete. Active nodes are mr01vm03 .

[root@mr01vm03 bin]# ./crsctl replace votedisk +DATAC3
 
Successful addition of voting disk 7493e8b7db5d4f5cbfe5a4113b260499.
Successful addition of voting disk 93a9cb21b7954fc5bf099cde51b6d71c.
Successful addition of voting disk c1026a740b244f12bf650d5727f5313d.
Successful deletion of voting disk 2d8f371e02d54ffabf33e189ed5c42c7.
Successful deletion of voting disk dbba5f46190a4f1dbf73753552e2e82b.
Successful deletion of voting disk d6caa92e49234f9ebfe6fe749ea7dac9.
Successfully replaced voting disk group with +DATAC3.
 
CW 12.2 log:
CRS-4266: Voting file(s) successfully replaced

[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.7;192.168.10.8/DATAC3_CD_02_mrceladm02; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.5;192.168.10.6/DATAC3_CD_02_mrceladm01; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1605: CSSD voting file is online: o/192.168.10.9;192.168.10.10/DATAC3_CD_02_mrceladm03; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1604: CSSD voting file is offline: o/192.168.10.7;192.168.10.8/RECOC3_CD_02_mrceladm02; details at (:CSSNM00069:) in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(127253)]CRS-1626: A Configuration change request completed successfully
[OCSSD(127253)]CRS-1601: CSSD Reconfiguration complete. Active nodes are mr01vm03 .

[root@mr01vm03 bin]# ./crsctl query css votedisk
 
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   7493e8b7db5d4f5cbfe5a4113b260499 (o/192.168.10.7;192.168.10.8/DATAC3_CD_02_mrceladm02) [DATAC3]
2. ONLINE   93a9cb21b7954fc5bf099cde51b6d71c (o/192.168.10.5;192.168.10.6/DATAC3_CD_02_mrceladm01) [DATAC3]
3. ONLINE   c1026a740b244f12bf650d5727f5313d (o/192.168.10.9;192.168.10.10/DATAC3_CD_02_mrceladm03) [DATAC3]
Located 3 voting disk(s).
 
[root@mr01vm03 bin]# ./crsctl stop crs
 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.crsd' on 'mr01vm03'
CRS-2677: Stop of 'ora.crsd' on 'mr01vm03' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.crf' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'mr01vm03'
CRS-2677: Stop of 'ora.drivers.acfs' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.crf' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.storage' on 'mr01vm03' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'mr01vm03'
CRS-2677: Stop of 'ora.mdnsd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.asm' on 'mr01vm03' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'mr01vm03'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'mr01vm03' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.evmd' on 'mr01vm03'
CRS-2677: Stop of 'ora.ctssd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.evmd' on 'mr01vm03' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'mr01vm03'
CRS-2677: Stop of 'ora.cssd' on 'mr01vm03' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'mr01vm03'
CRS-2673: Attempting to stop 'ora.gipcd' on 'mr01vm03'
CRS-2677: Stop of 'ora.gipcd' on 'mr01vm03' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'mr01vm03' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mr01vm03' has completed
 
CW 12.2 log:
CRS-4133: Oracle High Availability Services has been stopped.
[GPNPD(127143)]CRS-2329: GPNPD on node mr01vm03 shut down.
[MDNSD(127076)]CRS-5602: mDNS service stopping by request.
[MDNSD(127076)]CRS-8504: Oracle Clusterware MDNSD process with operating system process ID 127076 is exiting
[OCTSSD(127440)]CRS-2405: The Cluster Time Synchronization Service on host mr01vm03 is shutdown by user
[OCTSSD(127440)]CRS-8504: Oracle Clusterware OCTSSD process with operating system process ID 127440 is exiting
[OCSSD(127253)]CRS-1603: CSSD on node mr01vm03 has been shut down.
[OCSSD(127253)]CRS-1660: The CSS daemon shutdown has completed
[OCSSD(127253)]CRS-8504: Oracle Clusterware OCSSD process with operating system process ID 127253 is exiting
[ORAROOTAGENT(126973)]CRS-5822: Agent '/u01/app/12.2.0.1/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:1:6} in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ohasd_orarootagent_root.trc.

[root@mr01vm03 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.


CW 12.2 log:
[OHASD(132769)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 132769
[OHASD(132769)]CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
[OHASD(132769)]CRS-2112: The OLR service started on node mr01vm03.
[OHASD(132769)]CRS-1301: Oracle High Availability Service started on node mr01vm03.
[OHASD(132769)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
[ORAROOTAGENT(132854)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 132854
[CSSDAGENT(132870)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 132870
[CSSDMONITOR(132882)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 132882
[ORAAGENT(132866)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 132866
[ORAAGENT(132932)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 132932
[MDNSD(132950)]CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 132950
[EVMD(132954)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 132954
[GPNPD(133002)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 133002
[GPNPD(133002)]CRS-2328: GPNPD started on node mr01vm03.
[GIPCD(133059)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 133059
[CSSDMONITOR(133145)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 133145
[CSSDAGENT(133161)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 133161
[OCSSD(133176)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 133176
[OCSSD(133176)]CRS-1713: CSSD daemon is started in hub mode
[OCSSD(133176)]CRS-1707: Lease acquisition for node mr01vm03 number 1 completed
[OCSSD(133176)]CRS-1605: CSSD voting file is online: o/192.168.10.5;192.168.10.6/DATAC3_CD_02_mrceladm01; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(133176)]CRS-1605: CSSD voting file is online: o/192.168.10.7;192.168.10.8/DATAC3_CD_02_mrceladm02; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(133176)]CRS-1605: CSSD voting file is online: o/192.168.10.9;192.168.10.10/DATAC3_CD_02_mrceladm03; details in /u01/app/oracle/diag/crs/mr01vm03/crs/trace/ocssd.trc.
[OCSSD(133176)]CRS-1601: CSSD Reconfiguration complete. Active nodes are mr01vm03 .
[OCSSD(133176)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
[OCTSSD(133414)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 133414
[OCTSSD(133414)]CRS-2403: The Cluster Time Synchronization Service on host mr01vm03 is in observer mode.
[OCTSSD(133414)]CRS-2401: The Cluster Time Synchronization Service started on host mr01vm03.
[OCTSSD(133414)]CRS-2407: The new Cluster Time Synchronization Service reference node is host mr01vm03.
[OSYSMOND(133806)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 133806
[CRSD(133818)]CRS-8500: Oracle Clusterware CRSD process is starting with operating system process ID 133818
[OLOGGERD(133919)]CRS-8500: Oracle Clusterware OLOGGERD process is starting with operating system process ID 133919
[CRSD(133818)]CRS-1012: The OCR service started on node mr01vm03.
[CRSD(133818)]CRS-1201: CRSD started on node mr01vm03.
[ORAAGENT(134062)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 134062
[ORAROOTAGENT(134088)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 134088
[ORAAGENT(134062)]CRS-5011: Check of resource "dbm03" failed: details at "(:CLSN00007:)" in "/u01/app/oracle/diag/crs/mr01vm03/crs/trace/crsd_oraagent_oracle.trc"
[CRSD(133818)]CRS-2772: Server 'mr01vm03' has been assigned to pool 'Generic'.
[CRSD(133818)]CRS-2772: Server 'mr01vm03' has been assigned to pool 'ora.dbm03'.

[root@mr01vm03 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       mr01vm03                 STABLE
ora.DATAC1.dg
               OFFLINE OFFLINE      mr01vm03                 STABLE
ora.DATAC3.dg
               ONLINE  ONLINE       mr01vm03                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       mr01vm03                 STABLE
ora.RECOC1.dg
               OFFLINE OFFLINE      mr01vm03                 STABLE
ora.RECOC3.dg
               ONLINE  ONLINE       mr01vm03                 STABLE
ora.chad
               ONLINE  OFFLINE      mr01vm03                 STABLE
ora.net1.network
               ONLINE  ONLINE       mr01vm03                 STABLE
ora.ons
               ONLINE  ONLINE       mr01vm03                 STABLE
ora.proxy_advm
               OFFLINE OFFLINE      mr01vm03                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       mr01vm03                 169.254.47.1 192.168
                                                             .10.15 192.168.10.16
                                                             ,STABLE
ora.asm
      1        ONLINE  ONLINE       mr01vm03                 Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.dbm03.db
      1        ONLINE  OFFLINE      mr01vm03                 STARTING
ora.mgmtdb
      1        ONLINE  OFFLINE      mr01vm03                 Instance Shutdown,ST
                                                             ARTING
ora.mr01vm03.vip
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.qosmserver
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       mr01vm03                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       mr01vm03                 STABLE
--------------------------------------------------------------------------------
[root@mr01vm03 bin]#

PRCD-1229 : An attempt to access configuration of database version 12.1 differs from version 12.2

1. First case

The customer had the GI version 12.1 and DB 12.1 before upgrade.
They upgraded the GI to version 12.2.
The only GI was upgraded. Database is still 12.1 (no changes were made to DB).

Old GI HOME: /u01/app/12.1.0.2/grid
New GI HOME: /u01/app/12.2.0.1/grid
DB HOME: /u01/app/oracle/product/12.1.0.2/dbhome_1


The upgrade finished successfully.
The ASM + CW + DB are up-and-running.

$ ps -ef|grep lgwr
oracle 18769 1 0 11:15 ? 00:00:00 asm_lgwr_+ASM1
oracle 20881 1 0 11:15 ? 00:00:00 mdb_lgwr_-MGMTDB
oracle 20943 1 0 11:15 ? 00:00:00 ora_lgwr_dbm051 

But the error appeared if to run GI srvctl against database:

$ /u01/app/12.2.0.1/grid/bin/srvctl stop db -d dbm05
PRCD-1229 : An attempt to access configuration of database dbm05 was rejected because its version 12.1.0.2.0 differs from the program version 12.2.0.1.0. Instead run the program from /u01/app/oracle/product/12.1.0.2/dbhome_1.

$ /u01/app/12.2.0.1/grid/bin/srvctl status db -d dbm05
PRCD-1229 : An attempt to access configuration of database dbm05 was rejected because its version 12.1.0.2.0 differs from the program version 12.2.0.1.0. Instead run the program from /u01/app/oracle/product/12.1.0.2/dbhome_1.

So, customer cannot start or stop the database or to look the database configuration.




Solution:

The documentation says:
"Only use the version of SRVCTL that is provided with Oracle Database 12c ...
The version of SRVCTL must be the same as the version of the object being managed"

This means:

• To manage Clusterware entities use commands from Clusterware Home
• To manipulate Database entities use commands from Database Home.

The command issued from DB Home work well:
$ cd $ORACLE_HOME/bin
$ ./srvctl status db -d dbm05


2. Second case

After manual upgrade of database TMC from 11.2.0.4 to 12.2.0.1
OLD_OH=/u02/oracle/oracle_base/TMC/11204
NEW_OH=/u02/oracle/oracle_base/TMC/12201

was obtained the error

$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
PRCD-1229 : An attempt to access configuration of database tmc_exa1 was rejected because its version 11.2.0.4.0 differs from the program version 12.2.0.1.0. Instead run the program from /u02/oracle/oracle_base/TMC/12201.

 Why ? I'm using the right database binaries!

Let's check:

[oracle@exa1dbadm01:TMC1 ~]$ cd /u02/oracle/oracle_base/TMC/11204
[oracle@exa1dbadm01:TMC1 11204]$ cd bin
[oracle@exa1dbadm01:TMC1 bin]$ ./srvctl config db -d tmc_exa1
Database unique name: TMC_EXA1
Database name:
Oracle home: /u02/oracle/oracle_base/TMC/11204  <<< Let change the wrong OH
Oracle user: oracle
Spfile: +DATAC1/TMC_EXA1/spfile.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: TMC_EXA1
Database instances: TMC1,TMC2
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: pi_tmcclone,tmc_clone
Type: RAC
Database is administrator managed


[oracle@exa1dbadm01:TMC1 bin]$ ./srvctl modify db -d tmc_exa1 -o /u02/oracle/oracle_base/TMC/12201

[oracle@exa1dbadm01:TMC1 bin]$ ./srvctl config db -d tmc_exa1
Database unique name: TMC_EXA1
Database name:
Oracle home: /u02/oracle/oracle_base/TMC/12201 <<< Now is OK
Oracle user: oracle
Spfile: +DATAC1/TMC_EXA1/spfile.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: TMC_EXA1
Database instances: TMC1,TMC2
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: pi_tmcclone,tmc_clone
Type: RAC
Database is administrator managed




But error still persist:

[oracle@exa1dbadm01:TMC1 bin]$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
PRCD-1229 : An attempt to access configuration of database tmc_exa1 was rejected because its version 11.2.0.4.0 differs from the program version 12.2.0.1.0. Instead run the program from /u02/oracle/oracle_base/TMC/12201.


Clusterware don't give the point to glue:
[root@exa1dbadm01 ~]# crsctl stat res ora.tmc_exa1.db -p|grep -i vers
TYPE_VERSION=3.3

SOLUTION:  use NEW_OH/srvctl upgrade command !


$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl UPGRADE database -d tmc_exa1 -oraclehome /u02/oracle/oracle_base/TMC/12201

$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
Instance TMC1 is not running on node exa1dbadm01
Instance TMC2 is not running on node exa1dbadm02

$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl start db -d tmc_exa1
PRCR-1079 : Failed to start resource ora.tmc_exa1.db
CRS-2501: Resource 'ora.tmc_exa1.db' is disabled

$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl enable db -d tmc_exa1
$  /u02/oracle/oracle_base/TMC/12201/bin/srvctl start db -d tmc_exa1
$  /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
Instance TMC1 is running on node exa1dbadm01
Instance TMC2 is running on node exa1dbadm02