The hidden space in Exadata, part 1

The very few people are aware about unallocated (free) space on the local disks of DB nodes of Exadata.

Connect to db node, run the vgdisplay and look the "Free PE" line. I'll show the X2-2 example:

[root@dbm1db02 ~]# vgdisplay
  --- Volume group ---
  VG Name               VGExaDb
  System ID            
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  13
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               556.80 GB
  PE Size               4.00 MB
  Total PE              142541
  Alloc PE / Size       39424 / 154.00 GB
  Free  PE / Size       103117 / 402.80 GB


As you can see we have 103117 free Physical Extent (PE Size =  4.00 MB).

Let use it:

[root@dbm1db02 ~]# lvcreate -l 103117 -n LVStore VGExaDb
  Logical volume "LVStore" created

I created the  " LVStore" - new logical volume in "VGExaDb" disk group.

You can see it in another way:

[root@dbm1db02 ~]# ls -l /dev/VGExaDb/
lrwxrwxrwx 1 root root 28 Jun 11 22:30 LVDbOra1 -> /dev/mapper/VGExaDb-LVDbOra1
lrwxrwxrwx 1 root root 29 Jun 11 22:30 LVDbSwap1 -> /dev/mapper/VGExaDb-LVDbSwap1
lrwxrwxrwx 1 root root 28 Jun 11 22:30 LVDbSys1 -> /dev/mapper/VGExaDb-LVDbSys1
lrwxrwxrwx 1 root root 27 Aug  2 16:57 LVStore -> /dev/mapper/VGExaDb-LVStore

The disk group VGExaDb now have no free space:

[root@dbm1db02 ~]# vgdisplay
  --- Volume group ---
  VG Name               VGExaDb
  System ID            
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  14
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               556.80 GB
  PE Size               4.00 MB
  Total PE              142541
  Alloc PE / Size       142541 / 556.80 GB
  Free  PE / Size       0 / 0

Let use new volume:

[root@dbm1db02 ~]# mkfs -t ext3 /dev/mapper/VGExaDb-LVStore
 
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
52805632 inodes, 105591808 blocks
5279590 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3223 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    102400000

Writing inode tables: done                           
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information:
done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@dbm1db02 ~]#
[root@dbm1db02 ~]# mkdir /store
[root@dbm1db02 ~]# e2label /dev/mapper/VGExaDb-LVStore store
[root@dbm1db02 ~]# vi /etc/fstab
[root@dbm1db02 ~]# mount /store 
[root@dbm1db02 ~]# df -h /store
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVStore
                      397G  199M  377G   1% /store

As you can see the X2-2 users can have additional 377g as new file system.

On the X4-2 Exadata you will have about 1.5T of additional space:

[root@ed02dbadm01 ~]# vgdisplay
  --- Volume group ---
  VG Name               VGExaDb
  System ID            
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  9
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                5
  Open LV               4
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               1.63 TB
  PE Size               4.00 MB
  Total PE              428308
  Alloc PE / Size       428308 / 1.63 TB
  Free  PE / Size       0 / 0

[root@ed02dbadm01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVStore
                      1.5T 1017G  375G  74% /store


Or you can go other way and extend / or /u01 using this unallocated space.