ExaWatcher in X4-2 Exadata

The new old product appeared in the Exadata - ExaWatcher, documentation is in note 1617454.1.

It can monitor IB switches and databases.

The new path is like the old OSW path :  /opt/oracle.ExaWatcher

What it can monitor ?  These files are directories in the /opt/oracle.ExaWatcher/archive:

Diskinfo, 

IBCardInfo

IBprocs

Iostat

LGWR

Lsof

MegaRaidFW

Meminfo

Mpstat

Netstat

Ps

RDSinfo

Slabinfo

Top

Vmstat

ExaWatcher saves files in the / file system (/opt/oracle.ExaWatcher/archiv is the / FS), / size is 30g .
It runs these cmds respectively and fill the directories above :

iostat -t -x

cat /sys/class/net/ib0/carrier;/bin/cat /sys/class/net/ib1/carrier;/usr/sbin/ibstatus;/usr/bin/ibv_devinfo;/sbin/ifconfig -a | /bin/cat

/opt/oracle.ExaWatcher/LGWRExaWatcher.sh  -  делает cat/proc/$pid_LGWR/schedstat

top -b

vmstat

ps -eo flags,s,ruser,pid,ppid,c,psr,pri,ni,addr,sz,wchan,stime,tty,time,cmd

netstat -a -i -n; netstat -s; netstat -n -p -l

/opt/oracle.ExaWatcher/RDSinfoExaWatcher.sh 2>/dev/null

mpstat -P ALL

lsof +c0 -w +L -b -R -i; lsof +c0 -w +L -b -R -U; lsof +c0 -w +L1 -b -R

/opt/oracle.ExaWatcher/IBCardInfoExaWatcher.sh 2>/dev/null

cat /proc/meminfo

cat /proc/slabinfo

/opt/MegaRAID/MegaCli/MegaCli64 fwtermlog dsply -a0

There is difference between original OSW and Exadata versions (OSW and EW).
Original OSW store historical files in the *.dat, but the Exadata's version store files in bzip2 :

[root@ed02dbadm01 oracle.ExaWatcher]# ls -l archive/RDSinfo.ExaWatcher/
total 15224
-rw-r----- 1 root root   5136 Dec  8 10:08 2013_12_07_21_18_23_RDSinfoExaWatcher_db01.domain.mycompany.com.dat.bz2
-rw-r----- 1 root root   5438 Jan 23 16:02 2014_01_23_03_02_25_RDSinfoExaWatcher_db01.domain.mycompany.com.dat.bz2
-rw-r----- 1 root root   4998 Jan 23 17:02 
...
2014_01_31_16_49_45_RDSinfoExaWatcher_ed02dbadm01.distr.fors.ru.dat

But currenly writing file is *.dat.



How long  store the history:  looking the code
/opt/oracle.ExaWatcher/ExaWatcherCleanup.sh
I see that deleting is triggered when file system / is filled 80% . This code choose the one oldest file in each directory and delete it. And run this cycle again until / is 20% free.


The log for deleted files: /opt/oracle.ExaWatcher/log/OldFilesDeleted.log

 And the processes are:

[root@ed02dbadm01 ~]# ps -ef|grep -i ExaWatcher

root      12753      1  0 14:58 ?        00:00:00 /bin/bash ./ExaWatcher.sh --fromconf

root      16161  12753  0 14:59 ?        00:00:04 /usr/bin/perl /opt/oracle.ExaWatcher/ExecutorExaWatcher.pl /opt/oracle.ExaWatcher/ExaWatcher.execonf

root      16237  16161  0 14:59 ?        00:00:00 sh -c /usr/bin/iostat -t -x  5  720 2>/dev/null >> /opt/oracle.ExaWatcher/archive/Iostat.ExaWatcher/2014_01_24_14_59_00_IostatExaWatcher_ed02dbadm01.distr.fors.ru.dat

root      16259  16161  0 14:59 ?        00:00:00 sh -c /usr/bin/top -b -d 5 -n 720 2>/dev/null >> /opt/oracle.ExaWatcher/archive/Top.ExaWatcher/2014_01_24_14_59_00_TopExaWatcher_ed02dbadm01.distr.fors.ru.dat

root      16312  16161  0 14:59 ?        00:00:00 sh -c /usr/bin/mpstat -P ALL  5  720 2>/dev/null >> /opt/oracle.ExaWatcher/archive/Mpstat.ExaWatcher/2014_01_24_14_59_00_MpstatExaWatcher_ed02dbadm01.distr.fors.ru.dat

root      16557  16161  0 14:59 ?        00:00:00 sh -c /opt/oracle.ExaWatcher/ExaWatcherCleanup.sh 1386431083 1701963863 3600 /opt/oracle.ExaWatcher/archive/ 3145728 2>>/dev/null

root      16558  16557  0 14:59 ?        00:00:00 /bin/bash /opt/oracle.ExaWatcher/ExaWatcherCleanup.sh 1386431083 1701963863 3600 /opt/oracle.ExaWatcher/archive/ 3145728

root      30003  16161  0 15:33 ?        00:00:00 sh -c /usr/bin/vmstat  5  2 >> /opt/oracle.ExaWatcher/archive/Vmstat.ExaWatcher/2014_01_24_14_59_00_VmstatExaWatcher_ed02dbadm01.distr.fors.ru.dat

You can see the some input numbers for the cleaner:

ExaWatcherCleanup.sh 1386431083 1701963863 3600 /opt/oracle.ExaWatcher/archive/ 3145728 2>>/dev/null

I think first 2 are the StartTime and EndTime for this script. So the ExaWatcher is supposed to work in time between

[root@ed02dbadm02 oracle.ExaWatcher]# date --date @1386431083
Sat Dec  7 19:44:43 MSK 2013
[root@ed02dbadm02 oracle.ExaWatcher]# date --date @1701963863
Thu Dec  7 19:44:23 MSK 2023

3600 - is cycle length. CleanupInterval.
/opt/oracle.ExaWatcher/archive/ - place for historical files
3145728 - the maximal size of archive directory in KB. SpaceLimit.

OSWatcher and the Exadata (OSWbb is not supported on the Exadata)

Oracle has the OSWatcher (OSWbb) which make the snapshots of the server load (CPU load, Memory load, Disks load ...) and save these data into file.

In the Exadata we also have the OSWatcher, we can see its processes and it save files into /opt/oracle.oswatcher/osw.

In the OSWbb is the analyser(painter) utility - oswbba.jar.

But I didn't find oswbba.jar in the Exadata.

I downloaded the latest OSWbb, put the oswbba.jar into the Exadata and run it.

As a rresult i've got:

"ERROR. You do not have a compatible version of OSWatcher to use with oswbba.
This version of oswbba requires files collected by running OSWbb(6.0 and
higher). The version of you archive files are, Jan ,
which may be compatible with an earlier version of OSWg or oswbba."

Why? 

 The questions are:
- Why there is no appropriate the oswbba.jar in the Exadata ?
- How to update the OSWbb in the Exadata correctly ? 

- Why there is no instructions for OSWbb in the Exadata documentation ...
- Why full-stack patch don't contain the latest OSWbb ?

Answers here:

- Exadata contain its own OSW version which is not compatible to OSWbb.  

- OSWbb is for systems other than exadata. OSWBB is not supported on exadata. 

- OSWatcher for the exadata will be upgrade with image upgrade.

- There is no analyzer (painter) for Exadata OSW.

- OSWatcher Graph (OSWg) Does Not Work on Exadata System (Note 1344568.1)