ZDLRA.ra_install.jar. Exception in thread "main" java.lang.IllegalStateException: Already connected or RMAN kbhssrGetSbtParms:OCIStmtExecute

Installing backup module (ra_install.jar) i obtained one strange error :

-bash-4.2$ /oracle/app/db12c/OPatch/jre/bin/java -jar /oracle/zdlra/ra_install.jar -dbUser ravpc1 -dbPass welcome1 -host ed05-scan.52.s -port 1521 -servicename zdlra -walletDir $ORACLE_HOME/dbs/ra_wallet -configFile $ORACLE_HOME/dbs/rachordprd1.ora -libDir $ORACLE_HOME/lib
Recovery Appliance Install Tool, build 12.2.0.1.0DBBKPCSBP_2017-11-28
Recovery Appliance credentials are valid.
Recovery Appliance wallet created in directory /oracle/app/db12c/dbs/ra_wallet.
Recovery Appliance initialization file /oracle/app/db12c/dbs/rachordprd1.ora created.
Downloading Recovery Appliance Software Library from file ra_linux64.zip.
  Error: Library download failed. Please check your network connection to Oracle Public Cloud.
Retry download from Recovery Appliance.
Downloading Recovery Appliance Software Library from file ra_linux64.zip.
Exception in thread "main" java.lang.IllegalStateException: Already connected
        at sun.net.www.protocol.http.HttpURLConnection.setRequestProperty(HttpURLConnection.java:3014)
        at oracle.backup.util.FileDownload.addDigestAuthHeader(FileDownload.java:699)
        at oracle.backup.util.FileDownload.addHttpAuthHeader(FileDownload.java:166)
        at oracle.backup.util.FileDownload.addHttpAuthHeader(FileDownload.java:138)
        at oracle.backup.util.FileDownload.download(FileDownload.java:418)
        at oracle.backup.util.ConfigSetup.downloadLibrary(ConfigSetup.java:452)
        at oracle.backup.ba.install.BaConfig.downloadLibrary(BaConfig.java:352)
        at oracle.backup.ba.install.BaConfig.main(BaConfig.java:155)

All paramters are correct, we checked is many times.
We checked the network connectivity, checked the firewall, did nslookup for all hosts for direct and reverse zones: all is correct !

But rman backups were working some times!
Some times the backup finished successfully, but some times backup got the error:

For example:
RMAN> RUN
2> {
3> ALLOCATE CHANNEL ch1 DEVICE TYPE sbt_tape
4> PARMS='SBT_LIBRARY=/oracle/app/db12c/lib/libra.so,
5> ENV=(RA_WALLET=LOCATION=file:/oracle/zdlra CREDENTIAL_ALIAS=ed05-scan.52.s:1521/zdlra)'  FORMAT '%U_%d';
6> #backup current controlfile;
7> BACKUP INCREMENTAL LEVEL 1 DATABASE ;
8> }
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on ch1 channel at 05/15/2018 14:19:27
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27023: skgfqsbi: media manager protocol error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   KBHS-01409: Oracle error reported from Recovery Appliance while executing kbhssrGetSbtParms:OCIStmtExecute
ORA-24324: service handle not initialized
KBHS-01412: failed for connection RAVPC1 (@ed05-scan.52.s:1521/zdlra)



We decided to stop one of nodes.
And while 2nd node is stopped and 1st node is running this error disappeared.
When 2nd node is working and 1st node is down - we always get error.


Diagnostics:
SQL> select * from rai_host;

NODE_NAME
--------------------------------------------------------------------------------
ADMIN_IP_ADDRESS
--------------------------------------------------------------------------------
BACKUP_IP_ADDRESS
--------------------------------------------------------------------------------
REPLICATION_IP_ADDRESS
--------------------------------------------------------------------------------
ed05dbadm02
172.31.8.177
10.35.60.243

ed05dbadm01.distr.fors.ru
172.31.8.176
10.35.60.241


There is difference in host names.


And the solution is obvious:

$ sqlplus rasys/welcome1

SQL> update rai_host set NODE_NAME='ed05dbadm02.distr.fors.ru' where ADMIN_IP_ADDRESS='172.31.8.177';

1 row updated.

SQL> commit  ;

Commit complete.

SQL> select * from rai_host;

NODE_NAME
--------------------------------------------------------------------------------
ADMIN_IP_ADDRESS
--------------------------------------------------------------------------------
BACKUP_IP_ADDRESS
--------------------------------------------------------------------------------
REPLICATION_IP_ADDRESS
--------------------------------------------------------------------------------
ed05dbadm02.distr.fors.ru
172.31.8.177
10.35.60.243

ed05dbadm01.distr.fors.ru
172.31.8.176
10.35.60.241

ZDLRA. ra_install.jar. Exception in thread "main" java.sql.SQLException: ORA-28040

If you run the  Oracle Backup module ra_install.jar and got the message:
Exception in thread "main" java.sql.SQLException: ORA-28040: No matching authentication protocol
then you need to fresh Oracle ODBC driver inside ra_install.jar file.



My story:
$ /oracle/app/db12c/OPatch/jre/bin/java -jar /oracle/zdlra/ra_install.jar \
-dbUser ravpc1 -dbPass welcome1 -host ed05-scan.52.s -port 1521 -servicename zdlra \
-walletDir /oracle/zdlra -configFile /oracle/zdlra/wallet.ora -libDir $ORACLE_HOME/lib

Recovery Appliance Install Tool, build 12.2.0.1.0DBBKPCSBP_2017-11-28
Connection to Recovery Appliance Failed.
Check -host, -port, -servicename, -dbUser and -dbPass values
Exception in thread "main" java.sql.SQLException: ORA-28040: No matching authentication protocol
        at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
        at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:110)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:171)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:455)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:406)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
        at oracle.jdbc.driver.T4CTTIoauthenticate.receiveOsesskey(T4CTTIoauthenticate.java:306)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:465)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at oracle.backup.ba.install.BaConfig.validateBaAccount(BaConfig.java:211)
        at oracle.backup.ba.install.BaConfig.doBaConfig(BaConfig.java:169)
        at oracle.backup.ba.install.BaConfig.main(BaConfig.java:154)

Solution:
The *.jar file is actually the ZIP file, which contain some directories. For example ra_install.jar:


The ojdbc8.jar contain similar directories:

Did you notice the "oracle" directory on the both pictures ?





In order to renew ODBC inside jar file you need to download new ojdbc.jar and copy "oracle" directory over old "oracle" directory. Latest ojdbc.jar is here: http://www.oracle.com/technetwork/database/features/jdbc/jdbc-ucp-122-3110062.html
Download ojdbc7.jar or ojdbc8.jar.

Сopy the oracle directory from ojdbc.jar into ra_install.jar.
Don't copy META-INF.

And your problem is solved !
Try install backup module another time.

ZDLRA. Configuring Protected Database. Backup Module

Q: In order to install backup module using ra_install.jar i need the java on the server. But there is no java installed on my database server. Should I install the OS java OR it is possible to use any other java ? For example, java from ORACLE_HOME, from GRID home or from opatch directory ?




A: If there is no OS java I use any other java to run ra_install_jar and have no problem :)


Look:
[root@ed04dbadm01 ~]# find /u01/app -name java
/u01/app/agentHome/agent_13.2.0.0.0/oracle_common/jdk/jre/bin/java
/u01/app/agentHome/agent_13.2.0.0.0/oracle_common/jdk/bin/java
/u01/app/12.2.0.1/grid/OPatch/jre/bin/java
/u01/app/12.2.0.1/grid/jdk/jre/bin/java
/u01/app/12.2.0.1/grid/jdk/bin/java
/u01/app/18.0.0/grid/jdk/jre/bin/java
/u01/app/18.0.0/grid/jdk/bin/java
/u01/app/oracle/product/11.2.0.3/dbhome_1/owb/wf/java
/u01/app/oracle/product/11.2.0.3/dbhome_1/lib/java
/u01/app/oracle/product/11.2.0.3/dbhome_1/xdk/doc/java
/u01/app/oracle/product/11.2.0.3/dbhome_1/sqldeveloper/lib/java
/u01/app/oracle/product/11.2.0.3/dbhome_1/jdk/jre/bin/java
/u01/app/oracle/product/11.2.0.3/dbhome_1/jdk/bin/java
/u01/app/oracle/product/12.2.0.1/dbhome_1/OPatch/jre/bin/java
/u01/app/oracle/product/12.2.0.1/dbhome_1/xdk/doc/java
/u01/app/oracle/product/12.2.0.1/dbhome_1/jdk/jre/bin/java
/u01/app/oracle/product/12.2.0.1/dbhome_1/jdk/bin/java
/u01/app/oracle/product/12.1.0.2/dbhome_1/OPatch/jre/bin/java
/u01/app/oracle/product/12.1.0.2/dbhome_1/xdk/doc/java
/u01/app/oracle/product/12.1.0.2/dbhome_1/sqldeveloper/sqldeveloper/demo/migration/application/exampleSource/java
/u01/app/oracle/product/12.1.0.2/dbhome_1/jdk/jre/bin/java
/u01/app/oracle/product/12.1.0.2/dbhome_1/jdk/bin/java
/u01/app/oracle/product/18.1.0/dbhome_1/xdk/doc/java
/u01/app/oracle/product/18.1.0/dbhome_1/jdk/jre/bin/java
/u01/app/oracle/product/18.1.0/dbhome_1/jdk/bin/java

I tested different javas: from Grid home, from OEM agent, from Oracle Home 12.2 and java from OPatch directory - all these javas work well !


The version of the java must be 1.7+, not 1.5, not 1.6.

[root@ed04dbadm01 ~]# /u01/app/agentHome/agent_13.2.0.0.0/oracle_common/jdk/jre/bin/java -version
java version "1.7.0_111"
Java(TM) SE Runtime Environment (build 1.7.0_111-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.111-b13, mixed mode)

[root@ed04dbadm01 ~]# /u01/app/12.2.0.1/grid/OPatch/jre/bin/java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)[root@ed04dbadm01 ~]# /u01/app/12.2.0.1/grid/OPatch/opatch version
OPatch Version: 12.2.0.1.9

 
[root@ed04dbadm01 ~]# /u01/app/12.2.0.1/grid/jdk/jre/bin/java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

[root@ed04dbadm01 ~]# /u01/app/oracle/product/12.1.0.2/dbhome_1/jdk/bin/java -version
java version "1.6.0_75"
Java(TM) SE Runtime Environment (build 1.6.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 20.75-b01, mixed mode)


Or you can use the OS java:
$ which java
/usr/bin/java