19c RU Bugs Fixes

An year ago i've wrote the post "Time to upgrade to 19c"

Now it is the time to refresh the picture.

After the time new release have been issued, the question arises - is it time to move to the new release?
Is it good enough to work in production by now?
Usually users wait a while for a release to be "good enough" and for bugs to be fixed.

What is the criterion we may use to define whether a release is good and fine tuned enough?

As one of the criteria, we can take the number of bugs that are fixed in the each quarterly patch.
The logic here is: in the beginning, after a new release, a few people immediately start using it.
That's why the number of patches in the quarterly patch list is pretty small.
Gradually, the activity of users is growing, and then the number of identified problems/bugs is growing too. Therefore, the number of fixes in the patchset also increases.
After the time the software reaches a state where all the major bugs are fixed and the number of patches in the patchset decreases. Such an inflection point can be considered a stage of maturity of the software product.

The 19c Database Quarterly Bug Fixes:

                           Oct-20 Jan-21  Apr-21  Jul-21  Oct-21  Jan-22  Apr-22
                            19.9   19.10   19.11   19.12   19.13   19.14   19.15
Advance Queuing               4       6      11       9      10      12       6
ASM                          10      31      22      27      29      16      11
Buffer Cache Management       9      14      20      16      14       5       9
Generic                      77     160     215     120     117      66      73
High Availability            32      69     128      64      84      53      67
Diag                          1       2       1       2       1       1       2
Global Data Services                          1       1      10       0       2
Heterogenous Services         0       3       1       1       0       0       1
JVM                           0       0       0       0       0       0       0
Multitenant                   6      23      22       8      10       6       8
Net                           0       0       4       1       0       0       4
OLAP                          0       1       0       1       0       0       0
LOB                           1       1       4       1       2       1       0
Security                      7      11      15       3       8      10       3
Security Service              2       2       5       4       4       7       1
Sharding                      0       1       2       3      10       5       3
TableSpace management        16      31      36      22      12       7      10
Storage Server Service       13       6       5       2       9       3       4
Streams                       1      10       4       1       2       1       6
Transaction Mgmt              9      10      20       7       5       9       5
Universal Storage Mgmt        0       1       1       0       0       0       0
Utilities                     1      10      10       4      12       4       2
Virtual OS                   12      32      36      27      28      15      18
PL/SQL                        4       4      14       5       7       3       2
Manageability                 1      14      12       7       3       1       5
XML Utilities                 1       1       6       6       7       1       1
XML database                  3      19      25      17      24      13       4
Miscellaneous                29      47     114      59      86      65      33
                   SUM:     239     509     734     418     494     304     280
 

 

 

The 19c Grid Infrastructure Quarterly Bug Fixes:

                           Oct-20 Jan-21  Apr-21  Jul-21  Oct-21 Jan-22 Apr-22
                            19.9   19.10   19.11   19.12   19.13  19.14   19.15
Portable ClusterWare        124     195     139     132     65      61      82
Universal Storage Mgmt       26      30      48      41     41      43      26
Miscellaneous Issues          -       -       -       -     22      28      23
                     SUM:   150     225     187     173    128     132     131

:)

ORA-12152: TNS: Unable to send break message

 The new Exadata customer came with a problem "ORA-12152: TNS: Unable to send break message".  (What is the in-band breaking and out-of-band breaking we know from Tanel Poder's post:
https://tanelpoder.com/2008/02/05/oracle-hidden-costs-revealed-part-1/ ).

The customer say: 

"  We're testing upgrade to Oracle 19c from 11.2.0.4. And on 19c we got a problem with interrupting the connection between the application (pl/sql developer, sqlplus) on the client machine and the Oracle database: ORA-12152: TNS: Unable to send break message.

This problem is show stopper to upgrade to 19c!

The connection between the application (pl / sql developer, sqlplus) on the client machine and the process on the Oracle server is interrupted if the client application does not generate traffic for 60 minutes. The application is waiting for a response from the long procedure. After the procedure on the server is actually finished, the application is still waiting for the procedure to complete. When trying to interrupt the connection from the application side, we get ORA-12152: TNS: Unable to send break message (Cause: Unable to send break message).

Network engineers don't see any problems.

Simple test case show this problem:

begin
 dbms_lock.sleep(3590);
end;
/
Finished successully.
 

But

begin
 dbms_lock.sleep(3610);
end;
/
is finished unsuccessfully. 

 "

Thanks to the detailed description, the customer's problem became clear. It is not a In-Band or OOB breaking. It is actually Dead Connection Detection : DCD was enchanced in 12c to reduce detection time.

DCD is mechanism which allow the RDBMS server to check if the client is alive.

This feature is configured on server side using sqlnet.expire_time in sqlnet.ora. The probe packet is sent to the client side every sqlnet.expire_time minutes. If database server have got an error then client is dead and server can close this connection. In pre-12c releases this work was done by NS layer in SQL*Net . 

The 12c mechanism is intended to reduce the detection time and minimize load from RDBMS. This new mechanism is based on the TCP-keepalive property of the socket. With this approach TCP-keepalive probes are sent by OS after the connection has been idle for some time. Because these probes are implemented on the OS level then RDBMS rely on socket state (don't need send its own probes).

But in 12c we still able to use the sqlnet.expire_time.
After the customer have set sqlnet.expire_time=10 the error "ORA-12152: TNS: Unable to send break message" disappeared.

  

ORA-01405: fetched column value is NULL after upgrade from 12.1 to 18c

SYMPTOMs:

After upgrade 12.1.0.2 -> 18.6 we obtained at any SQL (select and DML) :

ORA-00604: error occurred at recursive SQL level 1
ORA-01405: fetched column value is NULL

At first (after successful upgrade) we see warning messages in alert log:

Completed: ALTER DATABASE OPEN /* db agent *//* {1:44916:15577} */
Unable to obtain current patch information due to error: 1405, ORA-01405: fetched column value is NULL
===========================================================
Dumping current patch information
===========================================================
Unable to obtain current patch information due to error: 1405
===========================================================
## jox_ujs_status: database not open read-write or Java not installed, returning FALSE in pid 258708

As you can see there is no expected patch list writen in alert log and issue concerning java.

Investigating the patch issue we found that all SQLs lead to ORA-1405 error:

SQL> select * from dual;
select * from dual
              *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-01405: fetched column value is NULL

We did the trace 10046 for the query above.
And for "select * from dual" we obtained simple trace file:

*** 2019-09-15T11:41:35.514839+03:00
WAIT #139918613321520: nam='SQL*Net message from client' ela= 4123724 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=5429869063221
CLOSE #139918613321520:c=5,e=5,dep=0,type=1,tim=5429869063321
=====================
PARSING IN CURSOR #139918613299752 len=102 dep=1 uid=0 oct=3 lid=0 tim=5429869064742 hv=3908278695 ad='27f813db8' sqlid='68hhnz3ng76d7'
select max_iops, max_mbps, max_pmbps, latency, num_disks, additional_info  from resource_io_calibrate$
END OF STMT
PARSE #139918613299752:c=399,e=966,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1003572626,tim=5429869064742
EXEC  #139918613299752:c=21,e=21,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1003572626,tim=5429869064821
FETCH #139918613299752:c=35,e=35,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=1003572626,tim=5429869064871
STAT  #139918613299752 id=1 cnt=1 pid=0 pos=1 obj=303 op='TABLE ACCESS STORAGE FULL RESOURCE_IO_CALIBRATE$ (cr=2 pr=0 pw=0 str=1 time=34 us cost=2 size=41 card=1)'
CLOSE #139918613299752:c=56,e=56,dep=1,type=0,tim=5429869064963
=====================
PARSE ERROR #139918613303136:len=18 dep=0 uid=0 oct=3 lid=0 tim=5429869064981 err=604
select * from dual
WAIT #139918613303136: nam='Disk file operations I/O' ela= 29 FileOperation=8 fileno=0 filetype=8 obj#=-1 tim=5429869065096
WAIT #139918613303136: nam='SQL*Net break/reset to client' ela= 2 driver id=1650815232 break?=1 p3=0 obj#=-1 tim=5429869065126
WAIT #139918613303136: nam='SQL*Net break/reset to client' ela= 63 driver id=1650815232 break?=0 p3=0 obj#=-1 tim=5429869065201
WAIT #139918613303136: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=5429869065217

*** 2019-09-15T11:41:37.746371+03:00
WAIT #139918613303136: nam='SQL*Net message from client' ela= 2229498 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=5429871294743
XCTEND rlbk=0, rd_only=1, tim=5429871294873
CLOSE #139918613303136:c=6,e=7,dep=0,type=0,tim=5429871294938


Nothing that would catch on the problem:  "PARSE ERROR ... err=604"

Security restrictions ? Broken audit triggers? VPD ?

None !

As it turned out, the message "ORA-01405: fetched column value is NULL" appears at intersection of:
-    Parallel SQL (parallel_degree_policy=AUTO)
-    parallel_degree_limit=IO 
-    upgrade 12.1 => 18c


REASON:

Oracle changed data format stored in the resource_io_calibrate$.ADDITIONAL_INFO column
and after the upgrade 18c database cannot recognise the content of this column.
As oracle say: " The DDL for resource_io_calibrate$ is different between 12c and 18.5.
the upgrade isn't handling the new ADDITIONAL_INFO column correctly"

So, if you use the parallel_degree_limit=IO + parallel_degree_policy=AUTO and there is data in the resource_io_calibrate$ table then can  obtain ORA-01405.

After we set parallel_degree_limit=CPU the error is disappeared.


SOLUTIONs:

- delete from resource_io_calibrate$;
  commit;
 And if you  need to populate resource_io_calibrate$ with new values, then: exec dbms_resource_manager.calibrate_io - .

  OR

- change the parallel_degree_limit to CPU
  (an i recommend next "delete from resource_io_calibrate$"  to avoid this error in the future)


Getting ORA-604, ORA-01405: fetched column value is NULL when PARALLEL_DEGREE_LIMIT = IO in 18c (Doc ID 2537431.1)

Upgrade GI from 12.2 to 18.4 on Virtual Exadata. Part 2.

Part 1: http://exadata-dba.blogspot.com/2018/12/upgrade-gi-from-122-to-184-on-virtual.html

1.

Read the documentation: https://docs.oracle.com/en/database/oracle/oracle-database/18/cwlin/upgrading-oracle-grid-infrastructure.html#GUID-DF76F201-3374-486F-9D19-06276764569F

18.1.0.0 Grid Infrastructure and Database Upgrade steps for Exadata Database Machine running 11.2.0.4 and later on Oracle Linux (Doc ID 2369422.1)

Patches to apply before upgrading Oracle GI and DB to 18c or downgrading to previous release (Doc ID 2414935.1)

2.

Download GI home and appropriate files:  GI base release, latest opatch, GI RU (p28689122_184000_Linux-x86-64.zip).

3.

Create a new Oracle Grid Infrastructure Oracle home. In this installation I suppose that GI files were unzipped at previous stem as root user in /u01/app/18c/grid file system, therefore we need to login to VM and chown -R oracle:oinstall /u01/app/18c/grid. In this environment GI and RDBMS software are under oracle user, so we have to change ownership of GI to  oracle:oinstall. Usually I prefer to have separate ownership: grid user for GI and oracle user for RDBMS , in this case you should chown grid:oinstall $GI_HOME.

As root@ DomU:

# mkdir -p /u01/app/18c/grid

# cat /etc/fstab

# echo "/dev/xvdg                /u01/app/18c/grid          ext4   defaults     1 1" >>/etc/fstab

# cat /etc/fstab

# mount /u01/app/18c/grid

# chown -R oracle:oinstall /u01/app/18c/grid

4.

Next step is to renew opatch software in GI home. The last opatch version for today is 16, and I renewed opatch simply unzipping it into GI home:

# su - oracle

$ unzip /store/p6880880_180000_Linux-x86-64.zip –d /u01/app/18c/grid
Archive:  /u01/opatch/p6880880_180000_Linux-x86-64.zip

replace /u01/app/18c/grid/OPatch/opatchprereqs/oui/knowledgesrc.xml? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

and answer A (overwrite allways).
The same opatch file is suit for GI and RDBMS homes.

5.

Next step is to apply latest GI RU patch (18.4 in my case) to the base release (18.3 in my case):

$ cd /u01/app/18c/grid

$ ./gridSetup.sh -silent -applyPSU /store/tmpEXAPSU/28689122/28659165

At this step I patched the base release binaries. Patch 18.4 adds about 2,5g to the GI_HOME.

6.

Next step is to edit the $GI_HOME/install/response/gridsetup.rsp file. I changed 2 empty lines. Before edit:
oracle.install.option=
ORACLE_BASE=

After edit:
oracle.install.option=UPGRADE
ORACLE_BASE=/u01/app/oracle

7.

The final step consist of run gridSetup.sh script.

$ ./gridSetup.sh  -silent -responseFile /u01/app/18c/grid/install/response/gridsetup.rsp

Launching Oracle Grid Infrastructure Setup Wizard...

In my 1^st run

$ ./gridSetup.sh  -silent -responseFile /u01/app/18c/grid/install/response/gridsetup.rsp

Launching Oracle Grid Infrastructure Setup Wizard...

I obtained the failed message:

[FATAL] [INS-13019] Some mandatory prerequisites are not met. These prerequisites cannot be ignored.

   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2018-12-20_11-51-23AM/gridSetupActions2018-12-20_11-51-23AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

The log file say that “ORACLE_BASE directory /u01/app/oracle is not writable”. I double checked writeability of ORACLE_BASE = /u01/app/oracle and found it writable. So I decided ignore this fatal error with –skipPrereqs option:

$ ./gridSetup.sh  -silent -responseFile /u01/app/18c/grid/install/response/gridsetup.rsp -skipPrereqs

Launching Oracle Grid Infrastructure Setup Wizard...

The response file for this session can be found at: /u01/app/18c/grid/install/response/grid_2018-12-20_11-55-36AM.rsp

You can find the log of this install session at: /u01/app/oraInventory/logs/GridSetupActions2018-12-20_11-55-36AM/gridSetupActions2018-12-20_11-55-36AM.log

As a root user, execute the following script(s):
        1. /u01/app/18c/grid/rootupgrade.sh

Execute /u01/app/18c/grid/rootupgrade.sh on the following nodes:
[var01vm03]

Successfully Setup Software.
As install user, execute the following command to complete the configuration.

        /u01/app/18c/grid/gridSetup.sh -executeConfigTools -responseFile /u01/app/18c/grid/install/response/gridsetup.rsp [-silent]

The gridSetup.sh completed successfully, so we should run next 2 steps:

1.  /u01/app/18c/grid/rootupgrade.sh

2.        gridSetup.sh –executeConfigTools …

8.

Next error becomes from roorupgrade.sh script:

[root@VM03]# /u01/app/18c/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/18c/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/18c/grid/crs/install/crsconfig_params

The log of current session can be found at:
  /u01/app/oracle/crsdata/var01vm03/crsconfig/rootcrs_var01vm03_2018-12-20_12-04-43AM.log

2018/12/20 12:04:45 CLSRSC-697: Failed to get the value of environment variable 'TZ' from the environment file '/u01/app/12.1.0.2/grid/crs/install/s_crsconfig_var01vm03_env.txt'

Died at /u01/app/18c/grid/crs/install/crsutils.pm line 17076.

There is no 12.1 GI in this configuration. It was removed a half year ago. So the simple solution – find tis file and temporary copy appropriate file to appropriate place:

[root@VM03]# find /u01 – name s_crsconfig_var01vm03_env.txt
/u01/app/12.2.0.1/grid/crs/install/s_crsconfig_var01vm03_env.txt

[root@VM03]# mkdir -p /u01/app/12.1.0.2/grid/crs/install/
[root@VM03]# cp /u01/app/12.2.0.1/grid/crs/install/s_crsconfig_var01vm03_env.txt /u01/app/12.1.0.2/grid/crs/install/

[root@VM03]# chown -R oracle:oinstall /u01/app/12.1.0.2

And run rootupgrade.sh one more time:

[root@VM03]# /u01/app/18c/grid/rootupgrade.sh

9.

Next step is to run executeConfigTools :

[oracle@VM03]$ /u01/app/18c/grid/gridSetup.sh -executeConfigTools -responseFile /u01/app/18c/grid/install/response/gridsetup.rsp -silent

10. MGMT DB

After upgrade we noticed that there is no MGMT DB in our GI 18.4. The investigation show that there were no MGMT DB in the previous 12.2 GI. So, the upgrade 12.2.0.1 -> 18c completed successfully while absence MGMT DB.

How to add the MGMT DB to GI 18.4:

/u01/app/18c/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName DATAC8

-datafileJarLocation /u01/app/18c/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords –skipUserTemplateCheck

$ /u01/app/18c/grid/bin/mgmtca –local

-------------------------------------------------------------------------------------

The log of rootupgrade.sh script, all 19 steps:

Performing root user operation.

The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /u01/app/18c/grid

   Copying dbhome to /usr/local/bin ...

   Copying oraenv to /usr/local/bin ...

   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Using configuration parameter file: /u01/app/18c/grid/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/oracle/crsdata/var01vm03/crsconfig/rootcrs_var01vm03_2018-12-20_12-14-20AM.log

2018/12/20 12:14:32 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.

2018/12/20 12:14:32 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2018/12/20 12:15:20 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2018/12/20 12:15:20 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.

2018/12/20 12:15:24 CLSRSC-595: Executing upgrade step 3 of 19: 'GetOldConfig'.

2018/12/20 12:15:24 CLSRSC-464: Starting retrieval of the cluster configuration data

2018/12/20 12:15:28 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.

2018/12/20 12:16:59 CLSRSC-693: CRS entities validation completed successfully.

2018/12/20 12:17:02 CLSRSC-515: Starting OCR manual backup.

2018/12/20 12:17:08 CLSRSC-516: OCR manual backup successful.

2018/12/20 12:17:13 CLSRSC-486:

 At this stage of upgrade, the OCR has changed.

 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2018/12/20 12:17:13 CLSRSC-541:

 To downgrade the cluster:

 1. All nodes that have been upgraded must be downgraded.

2018/12/20 12:17:13 CLSRSC-542:

 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2018/12/20 12:17:13 CLSRSC-615:

 3. The last node to downgrade cannot be a Leaf node.

2018/12/20 12:17:16 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2018/12/20 12:17:16 CLSRSC-595: Executing upgrade step 4 of 19: 'GenSiteGUIDs'.

2018/12/20 12:17:22 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.

2018/12/20 12:17:24 CLSRSC-363: User ignored prerequisites during installation

2018/12/20 12:17:31 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.

2018/12/20 12:17:36 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.

2018/12/20 12:17:36 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.

2018/12/20 12:18:34 CLSRSC-470: Starting non-rolling migration of Oracle ASM

2018/12/20 12:18:34 CLSRSC-482: Running command: '/u01/app/18c/grid/bin/asmca -silent -upgradeNodeASM -nonRolling true -oldCRSHome /u01/app/12.2.0.1/grid -oldCRSVersion 12.2.0.1.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2018/12/20 12:18:41 CLSRSC-471: Successfully initiated non-rolling migration of Oracle ASM

2018/12/20 12:18:43 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2018/12/20 12:19:25 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

2018/12/20 12:19:28 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.

2018/12/20 12:19:28 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.

2018/12/20 12:19:35 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.

2018/12/20 12:19:35 CLSRSC-595: Executing upgrade step 12 of 19: 'UpgradeAFD'.

2018/12/20 12:19:40 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.

2018/12/20 12:19:44 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.

2018/12/20 12:19:45 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

2018/12/20 12:20:16 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'var01vm03'

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'var01vm03' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

2018/12/20 12:20:54 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.

2018/12/20 12:21:18 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'var01vm03'

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'var01vm03' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.evmd' on 'var01vm03'

CRS-2672: Attempting to start 'ora.mdnsd' on 'var01vm03'

CRS-2676: Start of 'ora.mdnsd' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.evmd' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'var01vm03'

CRS-2676: Start of 'ora.gpnpd' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'var01vm03'

CRS-2676: Start of 'ora.gipcd' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'var01vm03'

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'var01vm03'

CRS-2676: Start of 'ora.cssdmonitor' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'var01vm03'

CRS-2672: Attempting to start 'ora.diskmon' on 'var01vm03'

CRS-2676: Start of 'ora.crf' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.diskmon' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.cssd' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'var01vm03'

CRS-2676: Start of 'ora.ctssd' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'var01vm03'

CRS-2676: Start of 'ora.asm' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'var01vm03'

CRS-2676: Start of 'ora.storage' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'var01vm03'

CRS-2676: Start of 'ora.crsd' on 'var01vm03' succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

CRS-6017: Processing resource auto-start for servers: var01vm03

CRS-2672: Attempting to start 'ora.scan3.vip' on 'var01vm03'

CRS-2672: Attempting to start 'ora.var01vm03.vip' on 'var01vm03'

CRS-2672: Attempting to start 'ora.scan1.vip' on 'var01vm03'

CRS-2672: Attempting to start 'ora.scan2.vip' on 'var01vm03'

CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'var01vm03'

CRS-2672: Attempting to start 'ora.ons' on 'var01vm03'

CRS-2676: Start of 'ora.MGMTLSNR' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.var01vm03.vip' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'var01vm03'

CRS-2676: Start of 'ora.scan3.vip' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'var01vm03'

CRS-2676: Start of 'ora.scan1.vip' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'var01vm03'

CRS-2676: Start of 'ora.scan2.vip' on 'var01vm03' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'var01vm03'

CRS-2676: Start of 'ora.LISTENER.lsnr' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.ons' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'var01vm03' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'var01vm03' succeeded

CRS-6016: Resource auto-start has completed for server var01vm03

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2018/12/20 12:22:43 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 12c Release 2.

Successfully taken the backup of node specific configuration in OCR.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

2018/12/20 12:23:00 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.

2018/12/20 12:23:03 CLSRSC-474: Initiating upgrade of resource types

2018/12/20 12:23:33 CLSRSC-475: Upgrade of resource types successfully initiated.

Start upgrade invoked..

2018/12/20 12:23:34 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded

2018/12/20 12:23:34 CLSRSC-482: Running command: '/u01/app/18c/grid/bin/crsctl set crs activeversion'

Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.

Started to upgrade CSS.

CSS was successfully upgraded.

Started to upgrade CRS.

CRS was successfully upgraded.

Successfully upgraded the active version of Oracle Clusterware.

Oracle Clusterware active version was successfully set to 18.0.0.0.0.

2018/12/20 12:24:36 CLSRSC-479: Successfully set Oracle Clusterware active version

2018/12/20 12:24:36 CLSRSC-476: Finishing upgrade of resource types

2018/12/20 12:24:37 CLSRSC-477: Successfully completed upgrade of resource types

2018/12/20 12:25:42 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.

2018/12/20 12:25:43 CLSRSC-476: Finishing upgrade of resource types

2018/12/20 12:25:44 CLSRSC-477: Successfully completed upgrade of resource types

2018/12/20 12:25:49 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded