You are here

10. Oracle VM NUMA Best Practices

10. Oracle VM NUMA Best Practices

Applicable to all Oracle VM 3.x Releases. 

Contemporary CPU’s from Intel and AMD have NUMA architectures. NUMA stands for Non-Uniform Memory Access. With NUMA each physical CPU (pCPU) will be assigned its own local memory. An assigned processor-memory pair is called a NUMA node. Local memory access from CPUs on the same socket will have significantly lower latency than remote memory access from CPUs on a different socket. 
 
Figure 52 shows the NUMA architecture.
Oracle VM NUMA Design
 
Oracle VM supports NUMA using a Xen feature called NUMA aware scheduling. NUMA aware scheduling will assign a virtual machine's vCPUs (virtual CPUs) to a NUMA node as a NUMA client. If a virtual machine has multiple vCPUs, the NUMA scheduler will always assign the virtual machine's vCPUs to a single NUMA node to maintain memory locality. For example, an Oracle Database virtual machine with 32 vCPUs allocated to a single NUMA node with 20 threads would be oversubscribed. CPU-bound workloads, such as Oracle Databases, should not be on Oracle VM Servers with oversubscribed CPUs. NUMA scheduling endeavors to keep vCPUs assigned to the same NUMA node to maintain memory locality.
 
If your supporting virtual machines with more vCPUs than its NUMA node, disable NUMA. For example, Xen NUMA aware scheduling will place a virtual machine with 32 vCPUs on a single NUMA node, even if the node does not have 32 cores or threads. 
The next example shows a single virtual machine with 32 vCPUs running on an Oracle VM Server with two NUMA CPUs. Each CPU has 10 cores with 20 threads, hyperthreading is enabled. Please note that NUMA scheduling has allocated the virtual machine's vCPUs to one of the NUMA nodes, and dom0 is allocated to the other NUMA node. The virtual machines vCPUs are oversubscribed by 12 CPUs. The only option to allocate 32 vCPUs to the following virtual machine is to disable NUMA. 
# xm vcpu-list 
Name                             ID  VCPU CPU State Time(s)  CPU Affinity 
0004fb000006000028e8d30845329a0c 4 	0  29 r--   799382.3 20-39 
0004fb000006000028e8d30845329a0c 4 	1  26 -b-   270889.4 20-39 
0004fb000006000028e8d30845329a0c 4 	2  27 -b-   268401.2 20-39 
0004fb000006000028e8d30845329a0c 4 	3  20 -b-   262000.4 20-39 
0004fb000006000028e8d30845329a0c 4 	4  36 -b-   268630.6 20-39 
0004fb000006000028e8d30845329a0c 4 	5  31 -b-   270950.8 20-39 
0004fb000006000028e8d30845329a0c 4 	6  23 -b-   259317.7 20-39 
0004fb000006000028e8d30845329a0c 4 	7  24 -b-   261852.4 20-39 
0004fb000006000028e8d30845329a0c 4 	8  33 r--   263039.4 20-39 
0004fb000006000028e8d30845329a0c 4 	9  37 -b-   255852.1 20-39 
0004fb000006000028e8d30845329a0c 4 	10 21 -b-   258479.8 20-39 
0004fb000006000028e8d30845329a0c 4 	11 27 -b-   260933.2 20-39 
0004fb000006000028e8d30845329a0c 4 	12 29 -b-   256616.4 20-39 
0004fb000006000028e8d30845329a0c 4 	13 28 -b-   253542.3 20-39 
0004fb000006000028e8d30845329a0c 4 	14 20 -b-   259498.4 20-39 
0004fb000006000028e8d30845329a0c 4 	15 34 -b-   259932.4 20-39 
0004fb000006000028e8d30845329a0c 4 	16 29 -b-   256240.4 20-39 
0004fb000006000028e8d30845329a0c 4 	17 37 -b-   250783.4 20-39 
0004fb000006000028e8d30845329a0c 4 	18 27 -b-   253684.5 20-39 
0004fb000006000028e8d30845329a0c 4 	19 26 -b-   263063.8 20-39 
0004fb000006000028e8d30845329a0c 4 	20 22 -b-   253006.8 20-39 
0004fb000006000028e8d30845329a0c 4 	21 39 r--   250176.9 20-39 
0004fb000006000028e8d30845329a0c 4 	22 32 -b-   251578.1 20-39 
0004fb000006000028e8d30845329a0c 4 	23 30 -b-   251288.6 20-39 
0004fb000006000028e8d30845329a0c 4 	24 24 -b-   255414.5 20-39 
0004fb000006000028e8d30845329a0c 4 	25 32 -b-   250540.4 20-39 
0004fb000006000028e8d30845329a0c 4 	26 25 -b-   255431.0 20-39 
0004fb000006000028e8d30845329a0c 4 	27 32 -b-   255604.3 20-39 
0004fb000006000028e8d30845329a0c 4 	28 27 -b-   254316.3 20-39 
0004fb000006000028e8d30845329a0c 4 	29 24 -b-   252208.0 20-39 
0004fb000006000028e8d30845329a0c 4 	30 39 -b-   243977.4 20-39 
0004fb000006000028e8d30845329a0c 4 	31 35 -b-   252555.0 20-39 
Domain-0 			 0 	0  0 -b-    429217.2 0 
Domain-0 			 0 	1  1 -b-    93104.6  1 
Domain-0 			 0 	2  2 -b-    53434.8  2 
Domain-0 			 0 	3  3 -b-    10838.6  3 
Domain-0 			 0 	4  4 -b-    6540.1   4 
Domain-0 			 0 	5  5 r--    7373.3   5 
Domain-0 			 0 	6  6 -b-    5934.0   6 
Domain-0 			 0 	7  7 -b-    7938.3   7 
Domain-0 			 0 	8  8 -b-    5966.3   8 
Domain-0 			 0 	9  9 -b-    7220.5   9 
Domain-0 			 0 	10 10 -b-   6257.7   10 
Domain-0 			 0 	11 11 -b-   7340.5   11 
Domain-0 			 0 	12 12 -b-   50423.1  12 
Domain-0 			 0 	13 13 -b-   7705.5   13 
Domain-0 			 0 	14 14 -b-   51551.9  14 
Domain-0 			 0 	15 15 -b-   11674.4  15 
Domain-0 			 0 	16 16 -b-   6063.5   16 
Domain-0 			 0 	17 17 r--   8334.1   17 
Domain-0 			 0 	18 18 -b-   5665.3   18 
Domain-0 			 0 	19 19 -b-   13142.2  19  
 
By default, most hardware will have NUMA enabled. The following example shows a 2 nodes NUMA Oracle VM Server:
# xm info -n
host                   : ovs-pickard
release                : 3.8.13-26.4.2.el6uek.x86_64
version                : #2 SMP Thu May 8 04:34:49 PDT 2014
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2133
hw_caps                : bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 73718
free_memory            : 45082
free_cpus              : 0
cpu_topology           : 
cpu:    core    socket     node
  0:       0        0        0
  1:       0        0        0
  2:       1        0        0
  3:       1        0        0
  4:       2        0        0
  5:       2        0        0
  6:       8        0        0
  7:       8        0        0
  8:       9        0        0
  9:       9        0        0
 10:      10        0        0
 11:      10        0        0
 12:       0        1        1
 13:       0        1        1
 14:       1        1        1
 15:       1        1        1
 16:       2        1        1
 17:       2        1        1
 18:       8        1        1
 19:       8        1        1
 20:       9        1        1
 21:       9        1        1
 22:      10        1        1
 23:      10        1        1
numa_info              : none
xen_major              : 4
xen_minor              : 3
xen_extra              : .1OVM
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 
xen_commandline        : console=com1,vga com1=57600,8n1 dom0_mem=max:2280M allowsuperpage dom0_vcpus_pin dom0_max_vcpus=20
cc_compiler            : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3)
cc_compile_by          : mockbuild
cc_compile_domain      : us.oracle.com
cc_compile_date        : Sun Oct 26 23:33:47 PDT 2014
xend_config_format     : 4
 
There are two option to disable NUMA. 
1) Disable NUMA in the server's BIOS or profile template. This is the preferred method to disable NUMA. 
2) Disable NUMA by editing the Oracle VM Server's grub.conf file. 
 
How to Disable NUMA in the Oracle VM Server's /boot/grub/grub.conf file
To disable NUMA in the Oracle VM Server's /boot/grub/grub.conf file, as root, edit /boot/grub/grub.conf and append parameter "numa=off" at the xen command line (xen.gz).
 
The next example shows a default Oracle VM Server Release 3.3.1 /boot/grub/grub.conf file. 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/sda2
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle VM Server-ovs (xen-4.3.0 3.8.13-26.4.2.el6uek.x86_64)
    root (hd0,0)
    kernel /xen.gz console=com1,vga com1=57600,8n1 dom0_mem=max:2280M allowsuperpage dom0_vcpus_pin dom0_max_vcpus=20
    module /vmlinuz-3.8.13-26.4.2.el6uek.x86_64 ro root=UUID=62493c65-273b-4688-987f-8ee99c5502b9 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
    module /initramfs-3.8.13-26.4.2.el6uek.x86_64.img
 
The next example shows the same /boot/grub/grub.conf file with the "numa=off" parameter appended on the xen command line (xen.gz).
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/sda2
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle VM Server-ovs (xen-4.3.0 3.8.13-26.4.2.el6uek.x86_64)
    root (hd0,0)
    kernel /xen.gz console=com1,vga com1=57600,8n1 dom0_mem=max:2280M allowsuperpage dom0_vcpus_pin dom0_max_vcpus=20 numa=off
    module /vmlinuz-3.8.13-26.4.2.el6uek.x86_64 ro root=UUID=62493c65-273b-4688-987f-8ee99c5502b9 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
    module /initramfs-3.8.13-26.4.2.el6uek.x86_64.img
 
The Oracle VM Server will need to be rebooted to disable NUMA. Next, confirm that NUMA is indeed disabled. As root, type the following command:
# xm info -n
host                   : ovs-pickard
release                : 3.8.13-26.4.2.el6uek.x86_64
version                : #2 SMP Thu May 8 04:34:49 PDT 2014
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 1
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2133
hw_caps                : bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 73718
free_memory            : 45082
free_cpus              : 0
cpu_topology           : 
cpu:    core    socket     node
  0:       0        0        0
  1:       0        0        0
  2:       1        0        0
  3:       1        0        0
  4:       2        0        0
  5:       2        0        0
  6:       8        0        0
  7:       8        0        0
  8:       9        0        0
  9:       9        0        0
 10:      10        0        0
 11:      10        0        0
 12:       0        1        0
 13:       0        1        0
 14:       1        1        0
 15:       1        1        0
 16:       2        1        0
 17:       2        1        0
 18:       8        1        0
 19:       8        1        0
 20:       9        1        0
 21:       9        1        0
 22:      10        1        0
 23:      10        1        0
numa_info              : none
xen_major              : 4
xen_minor              : 3
xen_extra              : .1OVM
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 
xen_commandline        : console=com1,vga com1=57600,8n1 dom0_mem=max:2280M allowsuperpage dom0_vcpus_pin dom0_max_vcpus=20
cc_compiler            : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3)
cc_compile_by          : mockbuild
cc_compile_domain      : us.oracle.com
cc_compile_date        : Sun Oct 26 23:33:47 PDT 2014
xend_config_format     : 4
 

Download PDF Now!