Tuesday, July 15, 2014

Exadata X4 multipathing with active-active infiniband

The X4 Exadata has the new feature "active-active" ports on the IB HCA.
As a result the v$asm_disk.PATH shows the 2 IP for the each disk path:

SQL> select path from v$asm_disk;

PATH
-------------------------------------------------------
o/192.168.11.9;192.168.11.10/RECOC1_CD_06_ed02celadm03
o/192.168.11.9;192.168.11.10/DATAC1_CD_09_ed02celadm03
o/192.168.11.9;192.168.11.10/DBFS_DG_CD_04_ed02celadm03
o/192.168.11.9;192.168.11.10/DBFS_DG_CD_03_ed02celadm03
o/192.168.11.9;192.168.11.10/DATAC1_CD_00_ed02celadm03
o/192.168.11.9;192.168.11.10/RECOC1_CD_02_ed02celadm03



[oracle@ed02dbadm01 ~]$ /sbin/ifconfig -a

ib0       Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.1 


ib1       Link encap:InfiniBand  HWaddr 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.2


As you can see above the DB node host 'ed02dbadm01' has the IPs 11.1 and 11.2, but disks have IPs 11.9 and 11.10. Therefore the disks are addressed with cell IP addresses, in our case 'ed02celadm03'. Let look its ifconfig:
[root@ed02celadm03 ~]# ifconfig -a

ib0       Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.9

ib1       Link encap:InfiniBand  HWaddr 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:192.168.11.10


Yes,

And let's look to cellip.ora on DB nodes, it now has 2 IPs because active-active ports (old version has 1 IP because active-passive ports):

[root@ed02dbadm01 ~]# cat /etc/oracle/cell/network-config/cellip.ora
cell="192.168.11.5;192.168.11.6"
cell="192.168.11.7;192.168.11.8"
cell="192.168.11.9;192.168.11.10"


No comments:

Post a Comment

cellcli -e list diskmap

The interesting LIST DISKMAP command in the storage cells links all views to hard disks: PCI bus address as Name 252:5, OS name (/dev/sdh,...