CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm')

 One day restarting clusterware i obtained CRS-6706:

[root@rac2 ~]# crsctl start cluster
CRS-6706: Oracle Clusterware Release patch level ('...') does not match Software patch level ('...').
Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

Ups!

Yes, i have patched the ClusterWare some days ago, but GI/CW successfully started after node reboot. I used the new Zero Downtime GI Patching procedure as described in great article of Daniel Overby Hansen: https://dohdatabase.com/2023/03/10/how-to-patch-oracle-grid-infrastructure-19c-using-zero-downtime-oracle-grid-infrastructure-patching

and may do something wrong ...  

At first I thought it was an error after the patching procedure.

I tried to start GI/CW from other node an obtained the same: "CRS-6706: Oracle Clusterware Release patch level ('...') does not match Software patch level ('...')". This confirmed my assumption of error is somewhere in the cluster software, maybe in cluster registry ...

Google was quick to suggest a suitable note with many clever recommendations:  CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm') (Doc ID 1639285.1) and some other interesting articles, they recommendto do:

$ORACLE_HOME/bin/clscfg -localpatch
$ORACLE_HOME/crs/install/rootcrs.sh -lock

[root@rac1 ~]# $ORACLE_HOME/bin/clscfg -localpatch
clscfg: EXISTING configuration version 0 detected.
Cannot proceed with the command execution while local OHASD is running.
Error initializing OLR for local patch operation.


[root@rac1 ~]# $ORACLE_HOME/crs/install/rootcrs.sh -lock
Using configuration parameter file: /u01/app/19.15/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac1/crsconfig/crslock_rac1_2023-07-26_06-30-09AM.log
Failure in execution (rc=-1, 0, No such file or directory) for command /usr/sbin/semanage fcontext --add -e /bin /u01/app/19.15/bin
2023/07/26 06:30:14 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

 Nothig helped !

I won't torture you with my investigation. The mistake is mine and it is very easy: the wrong value of ORACLE_HOME variable. :)

For convenience, I have specify a value for ORACLE_HOME in .bash_profile for user root in order to run GI commands from root user (don't setup GI variables each time i did "su - root"). And, of course, this variable was pointing to the wrong path:

Old GI home is /u01/app/19.15/grid
New GI home is /u01/app/19.16/grid

But my variable was ORACLE_HOME=/u01/app/19.15
Nothing more!

After obvious correction CW started successfully.

You may easy reproduce this error spoiling ORACLE_HOME ...