Tuesday, July 15, 2014

Exadata X4 multipathing with active-active infiniband

The X4 Exadata has the new feature "active-active" ports on the IB HCA.
As a result the v$asm_disk.PATH shows the 2 IP for the each disk path:

SQL> select path from v$asm_disk;

PATH
-------------------------------------------------------
o/192.168.11.9;192.168.11.10/RECOC1_CD_06_ed02celadm03
o/192.168.11.9;192.168.11.10/DATAC1_CD_09_ed02celadm03
o/192.168.11.9;192.168.11.10/DBFS_DG_CD_04_ed02celadm03
o/192.168.11.9;192.168.11.10/DBFS_DG_CD_03_ed02celadm03
o/192.168.11.9;192.168.11.10/DATAC1_CD_00_ed02celadm03
o/192.168.11.9;192.168.11.10/RECOC1_CD_02_ed02celadm03



[oracle@ed02dbadm01 ~]$ /sbin/ifconfig -a

ib0       Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.1 


ib1       Link encap:InfiniBand  HWaddr 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.2


As you can see above the DB node host 'ed02dbadm01' has the IPs 11.1 and 11.2, but disks have IPs 11.9 and 11.10. Therefore the disks are addressed with cell IP addresses, in our case 'ed02celadm03'. Let look its ifconfig:
[root@ed02celadm03 ~]# ifconfig -a

ib0       Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.9

ib1       Link encap:InfiniBand  HWaddr 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.10


Yes,

And let's look to cellip.ora on DB nodes, it now has 2 IPs because active-active ports (old version has 1 IP because active-passive ports):

[root@ed02dbadm01 ~]# cat /etc/oracle/cell/network-config/cellip.ora
cell="192.168.11.5;192.168.11.6"
cell="192.168.11.7;192.168.11.8"
cell="192.168.11.9;192.168.11.10"


No comments:

Post a Comment

Exadata account locked, pam_tally2 and host_access_control

Exam accounts are set up so that they are blocked for 10 minutes after the first wrong password entered. This brings a lot of inconvenien...