gridSetup.sh -applyRU ERROR: The home is not clean. This home cannot be used since there was a failed OPatch execution in this home. Use a different home to proceed.

I plan to install RAC VM on my notebook. I created new VM from Oracle Linux 7.7 and unzipped 19.3 Grid home and need to apply 19.7 RU.

During the gridSetup.sh -applyRU procedure we sometime may obtain a failing message:

$ ./gridSetup.sh -silent -applyRU /stage/30899722

ERROR: The home is not clean. This home cannot be used since there was a failed OPatch execution in this home. Use a different home to proceed.

And some bloggers may give you the advice: "When the command gridSetup.sh fails,  it makes the NEW_HOME unusable. The only way to get around this issue currently is to clean the contents out of that HOME , and unzip the gold image again. "

There is another solution: add the Grid home to oraInventory and use the opatch apply.

I have fresh Oracle Linux 7.7. VM and Grid 19.3 base image and 19.7 patch.

 

[grid@node1 grid]$ opatch lspatches
Argument(s) Error... Oracle Home's central inventory is not found.
Please check the arguments and try again.
OPatch failed with error code 135

This error because there is no Grid home in fresh VM.

So we need to register Grid home in  Central Inventory.

[grid@node1 grid]$ mkdir /u01/app/oraInventory
 

[grid@node1 grid]$ $OH/oui/bin/runInstaller -silent -attachHome ORACLE_HOME="/u01/app/19.0.0/grid" ORACLE_HOME_NAME="OraGI197Home1" CRS="true"
 

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5118 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
You can find the log of this install session at:
 /u01/app/oraInventory/logs/AttachHome2020-09-09_10-13-14PM.log
Please execute the '/u01/app/oraInventory/orainstRoot.sh' script at the end of the session.
'AttachHome' was successful.
 

 [grid@node1 30899722]$ pwd
/stage/30899722

[grid@node1 30899722]$ ll
total 132
drwxr-x--- 5 grid oinstall     81 Apr 11 01:16 30869156
drwxr-x--- 5 grid oinstall     62 Apr 11 01:18 30869304
drwxr-x--- 5 grid oinstall     62 Apr 11 01:19 30894985
drwxr-x--- 4 grid oinstall     48 Apr 11 01:18 30898856
drwxr-x--- 2 grid oinstall   4096 Apr 11 01:20 automation
-rw-rw-r-- 1 grid oinstall   5054 Apr 11 04:37 bundle.xml
-rw-r--r-- 1 grid oinstall 122266 Apr 11 04:25 README.html
-rw-r--r-- 1 grid oinstall      0 Apr 11 01:20 README.txt

[grid@node1 30898856]$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.21
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-09-12_15-42-19PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30898856

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/19.0.0/grid')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30898856' to OH '/u01/app/19.0.0/grid'

Patching component oracle.tomcat.crs, 19.0.0.0.0...
Patch 30898856 successfully applied.
Sub-set patch [29401763] has become inactive due to the application of a super-set patch [30898856].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-09-12_15-42-19PM_1.log

OPatch succeeded.

Next patch 30898856:

[grid@node1 30899722]$ ll
drwxr-x--- 5 grid oinstall     81 Apr 11 01:16 30869156
drwxr-x--- 5 grid oinstall     62 Apr 11 01:18 30869304
drwxr-x--- 5 grid oinstall     62 Apr 11 01:19 30894985
drwxr-x--- 4 grid oinstall     48 Apr 11 01:18 30898856
drwxr-x--- 2 grid oinstall   4096 Apr 11 01:20 automation
-rw-rw-r-- 1 grid oinstall   5054 Apr 11 04:37 bundle.xml
-rw-r--r-- 1 grid oinstall 122266 Apr 11 04:25 README.html
-rw-r--r-- 1 grid oinstall      0 Apr 11 01:20 README.txt
 

[grid@node1 30899722]$ pwd
/stage/30899722

[grid@node1 30899722]$ opatch lspatches
30898856;TOMCAT RELEASE UPDATE 19.0.0.0.0 (30898856)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)

OPatch succeeded.
 

[grid@node1 30899722]$ cd 30869304
[grid@node1 30869304]$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.21
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-09-12_16-17-11PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30869304

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/19.0.0/grid')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30869304' to OH '/u01/app/19.0.0/grid'

Patching component oracle.usm, 19.0.0.0.0...
Patch 30869304 successfully applied.
Sub-set patch [29517247] has become inactive due to the application of a super-set patch [30869304].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-09-12_16-17-11PM_1.log

OPatch succeeded.

[grid@node1 30869304]$ opatch lspatches
30869304;ACFS RELEASE UPDATE 19.7.0.0.0 (30869304)
30898856;TOMCAT RELEASE UPDATE 19.0.0.0.0 (30898856)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)

OPatch succeeded.

...

And after apply all 4 patches:

[grid@node1 30869156]$ opatch lspatches
30869156;Database Release Update : 19.7.0.0.200414 (30869156)
30894985;OCW RELEASE UPDATE 19.7.0.0.0 (30894985)
30869304;ACFS RELEASE UPDATE 19.7.0.0.0 (30869304)
30898856;TOMCAT RELEASE UPDATE 19.0.0.0.0 (30898856)

And the last step - Detach Grid home from Central Inventory:

[grid@node1 30869156]$ $OH/oui/bin/runInstaller -silent -detachHome ORACLE_HOME="/u01/app/19.0.0/grid" ORACLE_HOME_NAME="OraGI197Home1"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5118 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
You can find the log of this install session at:
 /u01/app/oraInventory/logs/DetachHome2020-09-12_04-33-16PM.log
'DetachHome' was successful.

$ opatch lspatches
Inventory load failed... LsPatchesSession::loadAndPrintInstalledPatch()
LsPatchesSession failed: OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.

OPatch failed with error code 2

I'm not sure the way described above is suppported solution.

The solution give us the Note Executing "gridSetup.sh" Fails with "ERROR: The home is not clean" (Doc ID 2279633.1)

ORA-12152: TNS: Unable to send break message

 The new Exadata customer came with a problem "ORA-12152: TNS: Unable to send break message".  (What is the in-band breaking and out-of-band breaking we know from Tanel Poder's post:
https://tanelpoder.com/2008/02/05/oracle-hidden-costs-revealed-part-1/ ).

The customer say: 

"  We're testing upgrade to Oracle 19c from 11.2.0.4. And on 19c we got a problem with interrupting the connection between the application (pl/sql developer, sqlplus) on the client machine and the Oracle database: ORA-12152: TNS: Unable to send break message.

This problem is show stopper to upgrade to 19c!

The connection between the application (pl / sql developer, sqlplus) on the client machine and the process on the Oracle server is interrupted if the client application does not generate traffic for 60 minutes. The application is waiting for a response from the long procedure. After the procedure on the server is actually finished, the application is still waiting for the procedure to complete. When trying to interrupt the connection from the application side, we get ORA-12152: TNS: Unable to send break message (Cause: Unable to send break message).

Network engineers don't see any problems.

Simple test case show this problem:

begin
 dbms_lock.sleep(3590);
end;
/
Finished successully.
 

But

begin
 dbms_lock.sleep(3610);
end;
/
is finished unsuccessfully. 

 "

Thanks to the detailed description, the customer's problem became clear. It is not a In-Band or OOB breaking. It is actually Dead Connection Detection : DCD was enchanced in 12c to reduce detection time.

DCD is mechanism which allow the RDBMS server to check if the client is alive.

This feature is configured on server side using sqlnet.expire_time in sqlnet.ora. The probe packet is sent to the client side every sqlnet.expire_time minutes. If database server have got an error then client is dead and server can close this connection. In pre-12c releases this work was done by NS layer in SQL*Net . 

The 12c mechanism is intended to reduce the detection time and minimize load from RDBMS. This new mechanism is based on the TCP-keepalive property of the socket. With this approach TCP-keepalive probes are sent by OS after the connection has been idle for some time. Because these probes are implemented on the OS level then RDBMS rely on socket state (don't need send its own probes).

But in 12c we still able to use the sqlnet.expire_time.
After the customer have set sqlnet.expire_time=10 the error "ORA-12152: TNS: Unable to send break message" disappeared.