Monday, September 16, 2019

ORA-01405: fetched column value is NULL after upgrade from 12.1 to 18c

SYMPTOMs:

After upgrade 12.1.0.2 -> 18.6 we obtained at any SQL (select and DML) :

ORA-00604: error occurred at recursive SQL level 1
ORA-01405: fetched column value is NULL

At first (after successful upgrade) we see warning messages in alert log:

Completed: ALTER DATABASE OPEN /* db agent *//* {1:44916:15577} */
Unable to obtain current patch information due to error: 1405, ORA-01405: fetched column value is NULL
===========================================================
Dumping current patch information
===========================================================
Unable to obtain current patch information due to error: 1405
===========================================================
## jox_ujs_status: database not open read-write or Java not installed, returning FALSE in pid 258708

As you can see there is no expected patch list writen in alert log and issue concerning java.

Investigating the patch issue we found that all SQLs lead to ORA-1405 error:

SQL> select * from dual;
select * from dual
              *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-01405: fetched column value is NULL

We did the trace 10046 for the query above.
And for "select * from dual" we obtained simple trace file:

*** 2019-09-15T11:41:35.514839+03:00
WAIT #139918613321520: nam='SQL*Net message from client' ela= 4123724 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=5429869063221
CLOSE #139918613321520:c=5,e=5,dep=0,type=1,tim=5429869063321
=====================
PARSING IN CURSOR #139918613299752 len=102 dep=1 uid=0 oct=3 lid=0 tim=5429869064742 hv=3908278695 ad='27f813db8' sqlid='68hhnz3ng76d7'
select max_iops, max_mbps, max_pmbps, latency, num_disks, additional_info  from resource_io_calibrate$
END OF STMT
PARSE #139918613299752:c=399,e=966,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1003572626,tim=5429869064742
EXEC  #139918613299752:c=21,e=21,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1003572626,tim=5429869064821
FETCH #139918613299752:c=35,e=35,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=1003572626,tim=5429869064871
STAT  #139918613299752 id=1 cnt=1 pid=0 pos=1 obj=303 op='TABLE ACCESS STORAGE FULL RESOURCE_IO_CALIBRATE$ (cr=2 pr=0 pw=0 str=1 time=34 us cost=2 size=41 card=1)'
CLOSE #139918613299752:c=56,e=56,dep=1,type=0,tim=5429869064963
=====================
PARSE ERROR #139918613303136:len=18 dep=0 uid=0 oct=3 lid=0 tim=5429869064981 err=604
select * from dual
WAIT #139918613303136: nam='Disk file operations I/O' ela= 29 FileOperation=8 fileno=0 filetype=8 obj#=-1 tim=5429869065096
WAIT #139918613303136: nam='SQL*Net break/reset to client' ela= 2 driver id=1650815232 break?=1 p3=0 obj#=-1 tim=5429869065126
WAIT #139918613303136: nam='SQL*Net break/reset to client' ela= 63 driver id=1650815232 break?=0 p3=0 obj#=-1 tim=5429869065201
WAIT #139918613303136: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=5429869065217

*** 2019-09-15T11:41:37.746371+03:00
WAIT #139918613303136: nam='SQL*Net message from client' ela= 2229498 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=5429871294743
XCTEND rlbk=0, rd_only=1, tim=5429871294873
CLOSE #139918613303136:c=6,e=7,dep=0,type=0,tim=5429871294938


Nothing that would catch on the problem:  "PARSE ERROR ... err=604"

Security restrictions ? Broken audit triggers? VPD ?

None !

As it turned out, the message "ORA-01405: fetched column value is NULL" appears at intersection of:
-    Parallel SQL (parallel_degree_policy=AUTO)
-    parallel_degree_limit=IO 
-    upgrade 12.1 => 18c


REASON:

Oracle changed data format stored in the resource_io_calibrate$.ADDITIONAL_INFO column
and after the upgrade 18c database cannot recognise the content of this column.
As oracle say: " The DDL for resource_io_calibrate$ is different between 12c and 18.5.
the upgrade isn't handling the new ADDITIONAL_INFO column correctly"

So, if you use the parallel_degree_limit=IO + parallel_degree_policy=AUTO and there is data in the resource_io_calibrate$ table then can  obtain ORA-01405.

After we set parallel_degree_limit=CPU the error is disappeared.


SOLUTIONs:

- delete from resource_io_calibrate$;
  commit;
 And if you  need to populate resource_io_calibrate$ with new values, then: exec dbms_resource_manager.calibrate_io - .

  OR

- change the parallel_degree_limit to CPU
  (an i recommend next "delete from resource_io_calibrate$"  to avoid this error in the future)


Getting ORA-604, ORA-01405: fetched column value is NULL when PARALLEL_DEGREE_LIMIT = IO in 18c (Doc ID 2537431.1)

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.

How to disable/setup autostart parameters for specified instance ?

Q: We have a 4-node RAC. I need to disable autostart of the DB on one node only.    How to do it and how to see autostart parameters, confir...