Saturday, August 11, 2018

10 Years of Exadata

ASM SGA sizing

The query from customer DBA :

" Hello,
We plan to install X TB database .. I didn't find the rule for ASM SGA sizing.

What values i need to set for the ASM instance:
sga_max_size =
sga_target     =   ? "


I found the note 437924.1 and 
the only source in the documentation, which point to the ASM sga sizing:




So, for example, for 1T database with Normal redundancy customer have to yield 24-25 MB of ASM SGA.


Friday, August 10, 2018

PRCD-1229 : An attempt to access configuration of database version 12.1 differs from version 12.2

1. First case

The customer had the GI version 12.1 and DB 12.1 before upgrade.
They upgraded the GI to version 12.2.
The only GI was upgraded. Database is still 12.1 (no changes were made to DB).

Old GI HOME: /u01/app/12.1.0.2/grid
New GI HOME: /u01/app/12.2.0.1/grid
DB HOME: /u01/app/oracle/product/12.1.0.2/dbhome_1


The upgrade finished successfully.
The ASM + CW + DB are up-and-running.

$ ps -ef|grep lgwr
oracle 18769 1 0 11:15 ? 00:00:00 asm_lgwr_+ASM1
oracle 20881 1 0 11:15 ? 00:00:00 mdb_lgwr_-MGMTDB
oracle 20943 1 0 11:15 ? 00:00:00 ora_lgwr_dbm051 

But the error appeared if to run GI srvctl against database:

$ /u01/app/12.2.0.1/grid/bin/srvctl stop db -d dbm05
PRCD-1229 : An attempt to access configuration of database dbm05 was rejected because its version 12.1.0.2.0 differs from the program version 12.2.0.1.0. Instead run the program from /u01/app/oracle/product/12.1.0.2/dbhome_1.


$ /u01/app/12.2.0.1/grid/bin/srvctl status db -d dbm05
PRCD-1229 : An attempt to access configuration of database dbm05 was rejected because its version 12.1.0.2.0 differs from the program version 12.2.0.1.0. Instead run the program from /u01/app/oracle/product/12.1.0.2/dbhome_1.

So, customer cannot start or stop the database or to look the database configuration.




Solution:

The documentation says:
"Only use the version of SRVCTL that is provided with Oracle Database 12c ...
The version of SRVCTL must be the same as the version of the object being managed"


This means:
  • To manage Clusterware entities use commands from Clusterware Home
  • To manipulate Database entities use commands from Database Home.

The command issued from DB Home work well:
$ cd $ORACLE_HOME/bin
$ ./srvctl status db -d dbm05


2. Second case

After manual upgrade of database TMC from 11.2.0.4 to 12.2.0.1
OLD_OH=/u02/oracle/oracle_base/TMC/11204
NEW_OH=/u02/oracle/oracle_base/TMC/12201

was obtained the error

$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
PRCD-1229 : An attempt to access configuration of database tmc_exa1 was rejected because its version 11.2.0.4.0 differs from the program version 12.2.0.1.0. Instead run the program from /u02/oracle/oracle_base/TMC/12201.


 Why ? I'm using the right database binaries!

Let's check:

[oracle@exa1dbadm01:TMC1 ~]$ cd /u02/oracle/oracle_base/TMC/11204
[oracle@exa1dbadm01:TMC1 11204]$ cd bin
[oracle@exa1dbadm01:TMC1 bin]$ ./srvctl config db -d tmc_exa1
Database unique name: TMC_EXA1
Database name:
Oracle home: /u02/oracle/oracle_base/TMC/11204  <<< Let change the wrong OH
Oracle user: oracle
Spfile: +DATAC1/TMC_EXA1/spfile.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: TMC_EXA1
Database instances: TMC1,TMC2
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: pi_tmcclone,tmc_clone
Type: RAC
Database is administrator managed



[oracle@exa1dbadm01:TMC1 bin]$ ./srvctl modify db -d tmc_exa1 -o /u02/oracle/oracle_base/TMC/12201

[oracle@exa1dbadm01:TMC1 bin]$ ./srvctl config db -d tmc_exa1
Database unique name: TMC_EXA1
Database name:
Oracle home: /u02/oracle/oracle_base/TMC/12201 <<< Now is OK
Oracle user: oracle
Spfile: +DATAC1/TMC_EXA1/spfile.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: TMC_EXA1
Database instances: TMC1,TMC2
Disk Groups: DATAC1,RECOC1
Mount point paths:
Services: pi_tmcclone,tmc_clone
Type: RAC
Database is administrator managed





But error still persist:

[oracle@exa1dbadm01:TMC1 bin]$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
PRCD-1229 : An attempt to access configuration of database tmc_exa1 was rejected because its version 11.2.0.4.0 differs from the program version 12.2.0.1.0. Instead run the program from /u02/oracle/oracle_base/TMC/12201.


Clusterware don't give the point to glue:
[root@exa1dbadm01 ~]# crsctl stat res ora.tmc_exa1.db -p|grep -i vers
TYPE_VERSION=3.3


SOLUTION:  use NEW_OH/srvctl upgrade command !


$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl UPGRADE database -d tmc_exa1 -oraclehome /u02/oracle/oracle_base/TMC/12201

$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
Instance TMC1 is not running on node exa1dbadm01
Instance TMC2 is not running on node exa1dbadm02


$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl start db -d tmc_exa1
PRCR-1079 : Failed to start resource ora.tmc_exa1.db
CRS-2501: Resource 'ora.tmc_exa1.db' is disabled


$ /u02/oracle/oracle_base/TMC/12201/bin/srvctl enable db -d tmc_exa1
$  /u02/oracle/oracle_base/TMC/12201/bin/srvctl start db -d tmc_exa1
$  /u02/oracle/oracle_base/TMC/12201/bin/srvctl status db -d tmc_exa1
Instance TMC1 is running on node exa1dbadm01
Instance TMC2 is running on node exa1dbadm02





Wednesday, August 8, 2018

Is it possible to use '-' sign (dash/hyphen/minus) in the hostname of Exadata ?

 The problem started from error:

CellCLI> list cell
     ed04celadm01     online

CellCLI> alter cell name = ed04-celadm01;
                                ^
CELL-01504: Invalid command syntax.

CellCLI> alter cell name = 'ed04-celadm01';

CELL-00012: Name value is not valid. ASCII characters a-zA-Z_0-9 required.

CellCLI> alter cell name="dc1-exa1celadm01";

CELL-00012: Name value is not valid. ASCII characters a-zA-Z_0-9 required.


So, the query is: is this possible to use '-' (dash/hyphen/minus) in the Exadata hostname or not ?

As shown above, Oracle prohibit to use dash in the name.

But, in other side, one our customer (large bank) has corporate standard, which require prefix in hostname: LOCATION-EXADATA#-HOSTNAME, for example: dc1-exa1-celadm01  (dc is datacenter name). And they used its corporate standard for many years and have no problems with hostname. And they bought the Exadata. And we need to run this machine.

The documentation say nothing about allowed characters in Exadata hostnames.

We checked: OEDA allowed the '-' sign in the prefix.
So me decided to install Exadata with dash in the hostname.
And installation was successful !

What the matter, what is the difference ?
 
The investigation shows that all hostnames have '-' in the name:
10.18.82.11 dc1-exa1-dbadm01 dc1-exa1-dbadm01.corp.icba.biz
10.18.82.12 dc1-exa1-dbadm02 dc1-exa1-dbadm02.corp.icba.biz
10.18.82.13 dc1-exa1-celadm01 dc1-exa1-celadm01.corp.icba.biz
10.18.82.14 dc1-exa1-celadm02 dc1-exa1-celadm02.corp.icba.biz
10.18.82.15 dc1-exa1-celadm03 dc1-exa1-celadm03.corp.icba.biz
10.18.82.21 dc1-exa1-sw-adm01 dc1-exa1-sw-adm01.corp.icba.biz
10.18.82.22 dc1-exa1-sw-iba01 dc1-exa1-sw-iba01.corp.icba.biz
10.18.82.23 dc1-exa1-sw-ibb01 dc1-exa1-sw-ibb01.corp.icba.biz
10.18.82.24 dc1-exa1-sw-pdua01 dc1-exa1-sw-pdua01.corp.icba.biz
10.18.82.25 dc1-exa1-sw-pdub01 dc1-exa1-sw-pdub01.corp.icba.biz


But CELL NAME was different !

CellCLI> list cell
       dc1_exa1_celadm01  online


Look: Oracle internally put the _ instead of -

And all celldisk and griddisk names are of the same style:


CellCLI> LIST CELLDISK
       CD_00_dc1_exa1_celadm01  normal
       CD_01_dc1_exa1_celadm01  normal
...
       FD_00_dc1_exa1_celadm01  normal
       FD_01_dc1_exa1_celadm01  normal
       FD_02_dc1_exa1_celadm01  normal
       FD_03_dc1_exa1_celadm01  normal

CellCLI> LIST GRIDDISK
       DATAC1_CD_00_dc1_exa1_celadm01  active
       DATAC1_CD_01_dc1_exa1_celadm01  active
...
       RECOC1_CD_00_dc1_exa1_celadm01  active
       RECOC1_CD_01_dc1_exa1_celadm01  active

...


Conclusions:
- It is safe to use '-' in the hostname of the Exadata.
- Divide the HOST name and CELL name (name in cellcli)
- The sign '-' take 1 place. As you should know the Grid Disk Name is limited by 23 chars. Additional chars lead to overallocation (exceeding this limit). So, i recommend - avoid '-' in the host names.


Thursday, July 19, 2018

Could not locate shrept.lst make: *** [client_sharedlib] Error 1

 Installing the quarterly database patch i got unpleasant message :

"
Patching component oracle.sdo, 12.2.0.1.0...
Make failed to invoke "/usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1"....'genclntsh: genclntsh: Could not locate  /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/shrept.lst
make: *** [client_sharedlib] Error 1
'
Make failed to invoke "/usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1"....'genclntsh: genclntsh: Could not locate  /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/shrept.lst
make: *** [client_sharedlib] Error 1
'
Make failed to invoke "/usr/bin/make -f ins_ldap.mk ldapsearch ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1"....'genclntsh: genclntsh: Could not locate  /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/shrept.lst
make: *** [/u01/app/oracle/product/12.2.0/dbhome_1/lib/libclntsh.so] Error 1
'

The following make actions have failed :

Re-link fails on target "client_sharedlib".
Re-link fails on target "client_sharedlib".
Re-link fails on target "ldapsearch".


Do you want to proceed? [y|n] y
User Responded with: Y
Patch 28163133 successfully applied.
Sub-set patch [27674384] has become inactive due to the application of a super-set patch [28163133].
Please refer to Doc ID 2161861.1 for any possible further required actions.
OPatch Session completed with warnings.
Log file location: /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatch/opatch2018-07-19_10-40-25AM_1.log

OPatch completed with warnings.
"


The reason of this error is obvious: i removed shrept.lst file from $OH/network/admin some time ago.

MOS note make advice to copy this file from similar $OH, but i have no similar $OH near me.

The solution was found to extract the shrept.lst  file from Database Installation Media file, in my case it is linuxx64_12201_database.zip .

1. Unzip linuxx64_12201_database.zip and go to directory
database/stage/Components/oracle.rdbms.rsf/12.2.0.1.0/1/DataFiles/

Here are 3 files :


 
2. The jar - is a zip-archive, so we can unzip it and search the shrept.lst in it
[oracle@oel6 DataFiles]$ unzip filegroup1.jar filegroup2.jar filegroup3.jar
[oracle@oel6 DataFiles]$ find . -name shrept*
./plsql/admin/shrept.lst
./shrept.lst
./rdbms/admin/shrept.lst

[oracle@oel6 DataFiles]$ ls -l ./plsql/admin/shrept.lst ./shrept.lst ./rdbms/admin/shrept.lst
-rw-r--r-- 1 oracle oinstall    72 Apr 15  1998 ./plsql/admin/shrept.lst
-rw-r--r-- 1 oracle oinstall 50099 May  5  2016 ./rdbms/admin/shrept.lst
-rwxr-x--- 1 oracle oinstall 50099 Jul 19 11:22 ./shrept.lst

The plsql is not our case.
It seems both last 2 files are the same. So make the copy and relink:
$ cp ./shrept.lst $OH/network/admin
$ cd $OH/bin
$ ./relink all

And now relink work well !
:)

Oracle Universal Installer... Checking swap space: 493 MB available, 500 MB required. Failed

After installing the patch i decided to relink all binaries. The relinking procedure is very simple:

$ cd $OH/bin
$ ./relink all 

But in my environment i got a unpleasant message:

" Starting Oracle Universal Installer...

Checking swap space: 493 MB available, 500 MB required.    Failed <<<<

Some requirement checks failed. You must fulfill these requirements before
continuing with the installation,

Exiting Oracle Universal Installer ...  "


Because I'm on the Linux, then the fastest solution is to make additional swap file.
It can be done online:

# free
# df -h
# dd if=/dev/zero of=/swapfile bs=1M count=100
# chmod 0600 /swapfile
# mkswap /swapfile
# swapon /swapfile
# df -h
# free


And relink  say:

"Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 593 MB    Passed
Preparing to launch Oracle Universal Installer ..."



To revert changes (online): 
#  swapoff /swapfile
#  rm /swapfile






Thursday, July 5, 2018

12.2+ SQL*Net fetures applicable for ExaStack

In 12.2 Oracle added ExaDirect SQL*Net Adapter:



 And in 18.1 Oracle deprecated SDP protoco (deprecated is supported, but not recommended):





https://docs.oracle.com/en/database/oracle/oracle-database/18/netag/changes-in-this-release.html#GUID-606D7C27-FBE9-47E7-B5A7-308CB576955E 
It is difficult to understand that Oracle:


https://docs.oracle.com/en/database/oracle/oracle-database/18/netag/understanding-communication-layers.html#GUID-E90842A7-D1ED-4356-91C7-E376E2889920

How to setup ExaDirect :
https://docs.oracle.com/en/database/oracle/oracle-database/12.2/netag/optimizing-performance.html#GUID-5499D049-CEB6-4AD1-99B6-0211D391908E

# ocrconfig -add +DATA PROT-30: The Oracle Cluster Registry location to be added is not usable. PROC-50: The Oracle Cluster Registry locatio...