You are here

Oracle VM Fault Testing

This chapter of the Oracle Cloud Cookbook describes how to fault test Oracle VM. This chapter applies to all Oracle VM releases.
Download PDF Now!
Before an Oracle VM server pool is placed into production, both network and storage fault tests should be conducted on each Oracle VM server to find compatible o2cb timeout values, an 802.3AD bond mode (1, 4 or 6) and network and storage configurations that provide predicable failure response. For example, Oracle VM Servers should be able to lose a bond port, redundant network or storage switch and/or an HBA without node evictions. Incompatible o2cb timeout values, 802.3AD bond modes, network switch and storage configurations can trigger node evictions and unexpected server reboots.
 
Tip: The only way to know if an Oracle VM server pool is production ready, is to fault test it. 
 

Oracle VM Architectural Review and Oracle VM Fault Testing

Oracle VM uses OCFS2 to manage up to 32 clustered Oracle VM Servers in an Oracle VM server pool. OCFS2 monitors the status of each server pool member using a network and storage heartbeat. If a server pool member fails to update or respond to network and/or storage heartbeats, the server pool member is fenced from the pool, promptly reboots, then all HA-enabled virtual machines are restarted on a live node in the pool. Fencing forcefully removes dead servers from a pool to ensure that active servers are not obstructed from accessing fenced servers cluster resources. The term “node eviction” is also used to describe servers fencing and reboots. A best practice is to design Oracle VM Server pools with dedicated network and storage network channels to avoid contention and unexpected server reboots.
 
A slightly modified version of OCSF2 (o2dlm) is bundled with Oracle VM. The OCFS2 file system and cluster stack are installed and configured as part of an Oracle VM Server installation. The o2cb service manages the cluster stack and the ocfs2 service manages the OCSF2 file system. The o2cb cluster service is a set of modules and in-memory file systems that manage the ocfs2 file system service, network and storage heartbeats and node evictions.
 
The Oracle Cluster File System 2 (OCFS2) is a general-purpose journaling file system developed by Oracle. Oracle released OCFS2 under the GNU General Public License (GPL), version 2. The OCSF2 source code and its tool set are part of the mainline Linux 2.6 kernel and above. The OCSF2 source code and its tool set can be downloaded from kernel.org, the Oracle Public Yum Server and from the Unbreakable Linux Network.
 
Oracle VM facilitates centralized server pool management using an agent-based architecture. The Oracle VM agent is a python application that is installed by default with Oracle VM Server. Oracle VM Manager dispatches commands using XML RPC over a dedicated network called the Server Management network channel using TCP/8899 to each server pool's Master Server agent. Each Master Server agent dispatch commands to subordinate agent servers using TCP/8899. The Oracle VM agent is also responsible for propagating the /etc/ocfs2/cluster.conf file to subordinate agent servers. There is only one Master Server in a server pool at any one point in time. The Master Server is the only server in a server pool to communicate with Oracle VM Manager.
 
Note: When Oracle VM Server is installed, the IP address entered during the installation is assigned to the Server Management network channel.
 
Once an Oracle VM server pool is created, two cluster configuration files are shared across all nodes in the server pool that maintain the cluster layout and cluster timeouts configurations. The /etc/ocfs2/cluster.conf file maintains the cluster layout and the /etc/sysconfig/o2cb file maintains the cluster timeouts. Both configuration files are read by the user-space utility configfs. configfs communicates the list of nodes in the /etc/ocfs2/cluster.conf file to the in-kernel node manager, along with the resource used for the heartbeat to the in-kernel heartbeat thread.
 
An Oracle VM server must be online to be a member of an Oracle VM pool/cluster. Once the cluster is on-line, each pool member starts a process, o2net. The o2net process creates TCP/IP intra-cluster node communication channels on port 7777 and sends regular keepalive packages to each node in the cluster to validate if the nodes are alive. The intra-cluster node communication uses the Cluster Heartbeat network channel. If a pool member loses network connectivity the keepalive connection becomes silent causing the node to self-fence. The keepalive connection time out value is managed in each nodes /etc/sysconfig/o2cb file's O2CB_IDLE_TIMEOUT_MS setting.
 
Along with the keepalive packages that check for node connectivity, the cluster stack also employs a disk heartbeat check. o2hb is the process that is responsible for the disk heartbeat component of cluster stack that actively monitors the status of all pool members. The heartbeat system uses a file on the OCSF2 file system, that each pool member periodically writes a block to, along with a time stamp. The time stamps are read by each pool member and are used to check if a pool member is alive or dead. If a pool member’s block stops getting updated, the node is considered dead, and self-fences. The disk heartbeat time out value is managed in each nodes /etc/sysconfig/o2cb file's O2CB_HEARTBEAT_THRESHOLD setting.
 
The OCFS2 network and storage heartbeat time out values are managed in each Oracle VM Servers /etc/sysconfig/o2cb file. Each pool member must have the same /etc/sysconfig/o2cb values. The default timeout values should be tested and tuned to your network and storage infrastructure to provide predicable failure response.
 
Tip: If a SAN storage controller fail over takes 120 seconds, and OCFS2 is set to the default value of 60 seconds, Oracle VM Servers will reboot halfway through the controller fail over. The O2CB_HEARTBEAT_THRESHOLD timeout value must longer then the SAN storage controller fail over timeout value.
 
The next example shows the default Oracle VM 3.2.x /etc/sysconfig/o2cb timeout values.
O2CB_IDLE_TIMEOUT_MS=60000 (60 secs)
O2CB_HEARTBEAT_THRESHOLD=91 (180 secs)
O2CB_RECONNECT_DELAY_MS=2000 (2 secs)
O2CB_KEEPALIVE_DELAY_MS=2000 (2 secs)
 
The next list explains each Oracle VM 3.2.x o2cb timeout value.
1- O2CB_IDLE_TIMEOUT_MS: Default settings is 60000 = 60 secs
Time in ms before a network connection is considered dead.
2- O2CB_HEARTBEAT_THRESHOLD: Default 91 = 180 secs
The disk heartbeat timeout is the number of two-second iterations before a node is considered dead. The exact formula used to convert the timeout in seconds to the number of iterations is:
O2CB_HEARTBEAT_THRESHOLD = (((timeout in seconds) / 2) + 1)
For example, to specify a 60 sec timeout, set it to 31. For 120 secs, set it to 61. The default for Oracle VM 3.2.x is 91 (180 secs), the default for Oracle VM 3.1.x is 31 (60 secs). 
Note: The O2CB_HEARTBEAT_THRESHOLD should be configured using the Oracle VM Manager GUI. The max setting for 3.2.x and above in Oracle VM Manager is 300, i.e. O2CB_HEARTBEAT_THRESHOLD = 151 in /etc/sysconfig/o2cb.
3- O2CB_RECONNECT_DELAY_MS: Default 2000 = 2 secs
Min time in ms between connection attempts
4- O2CB_KEEPALIVE_DELAY_MS: Default 2000 = 2 secs
Max time in ms before a keepalive packet is sent
 
A minimum of one Ethernet network interface (NIC) card is required to install Oracle VM. We recomend at least four 10G NICs configured with two bonds using tagged vlans, or five or more vNICs (Cisco UCS) using access ports. Trunk ports/802.1Q and/or access ports with NIC bonding mode 1, 4 and 6 are supported and configured post Oracle VM Server installation with Oracle VM Manager and/or Oracle Enterprise Manager. Mode 1 and 6 do not require special network switch settings. Mode 4 requires the network switch to support 802.3ad/LACP. Mode 6 bonds with the tagged vlans are supported only up to Oracle VM Release 3.3.1..
 
Network Switch Options
Trunk ports and/or Access ports options:
  • An access port is limited to one vlan per port.
  • A trunk port can have two or more vlans per port.
Most network switches support the following standards: EtherChannel, Port Channels, Link Aggregation Control Protocol (LACP) / 802.3ad, etc... Please consult Cisco for network switch configuration details: http://www.cisco.com/en/US/tech/tk389/tk213/tsd_technology_support_proto...
 
If using NIC bonding, each bond should be fault tested with various bond modes and network switch settings to confirm which NIC bonding mode and network switch setting combination provides the best throughput and predicable failure response. List 1 shows each of the Oracle VM network channels, NIC bonding modes, and a variety of network switch options that could be fault tested.
 
  • iLO (Integrated Lights Out Manager)
    • iLO ports enable browser based access to servers for installations and management. iLO ports should be on a network isolated from the server payload networks.
    • Network Class: A, B or C
    • Note: iLo is not managed by Oracle VM Manager. ilo is included in this table for completeness.
  • Server Management
    • The Server Management network is the communication channel for Oracle VM Manager, Oracle VM Server agents as well as administrative ssh access to Oracle VM Servers, and http/https/VNC access to and from Oracle VM Manager. Oracle VM Manager dispatches commands using XML RPC over the Server Management network using TCP/8899 to each server pool's master agent server. Each Master Server agent server dispatch commands to subordinate agent servers using the Server Management network with TCP/8899.
    • The Server Management network channel should be deployed on an a routable network. Many of our customers deploy the Server Management network channel on their public network. 
    • Network Class: A, B or C
    • Server/Network Switch Options: Trunk ports/802.1Q and/or access ports with NIC bonding mode 1, 4 and 6. Mode 1 and 6 do not require special network switch settings. Mode 4 requires the network switch to support 802.3ad/LACP. Mode 6 bonds with the tagged vlans are supported only up to Oracle VM Release 3.3.1..
  • Cluster Heartbeat
    • An Oracle VM server must be online to be a member of an Oracle VM pool/cluster. Once the cluster is on-line, each pool member starts a process, o2net. The o2net process creates TCP/IP intra-cluster node communication channels on port 7777 and sends regular keepalive packages to each node in the cluster to validate if the nodes are alive. The intra-cluster node communication uses the Cluster Heartbeat network channel. If a pool member loses network connectivity the keepalive connection becomes silent causing the node to self-fence. The keepalive connection time out value is managed in each nodes /etc/sysconfig/o2cb file's O2CB_IDLE_TIMEOUT_MS setting.
    • The Cluster Heartbeat network channel should be deployed on a dedicated private non-routable network.
    • Network Class: A, B or C
    • Server/Network Switch Options: Trunk ports/802.1Q and/or access ports with NIC bonding mode 1, 4 and 6. Mode 1 and 6 do not require special network switch settings. Mode 4 requires the network switch to support 802.3ad/LACP. Mode 6 bonds with the tagged vlans are supported only up to Oracle VM Release 3.3.1..
  • Live Migration
    • The Oracle VM Live Migration feature moves running virtual machines between server pool members across a LAN without loss of availability. Oracle VM uses an iterative precopy method to migrate running virtual machines between two pool members over the Live Migration network channel. A Live Migration event starts when the source server sends a migration request to the target server, which contains the virtual machines resource requirements. If the target accepts the migration request, the source starts the iterative precopy phase. The iterative precopy phase starts by iteratively copying the guest’s memory pages from the source to the target server over the Live Migration network channel. If a memory page changes during the precopy phase, it is marked dirty and resent. Once the majority of the pages are copied, the stop-and-copy phase begins. The stop-and-copy phase starts by pausing the guest while the remaining dirty pages are copied to the target, which usually takes 60 to 300 milliseconds. Once the pages are copied to the target, the virtual machine is started on target server.
    • The Live Migration network channel should be deployed on a private non-routable network.
    • Network Class: A, B or C
    • Server/Network Switch Options: Trunk ports/802.1Q and/or access ports with NIC bonding mode 1, 4 and 6. Mode 1 and 6 do not require special network switch settings. Mode 4 requires the network switch to support 802.3ad/LACP. Mode 6 bonds with the tagged vlans are supported only up to Oracle VM Release 3.3.1..
  • Storage
    • The Storage network channel is only used for iSCI and NFS storage. FC/SAN use a dedicated fibre fabric.
    • Network Class: A, B or C
    • Server/Network Switch Options: Trunk ports/802.1Q and/or access ports with NIC bonding mode 1, 4 and 6. Mode 1 and 6 do not require special network switch settings. Mode 4 requires the network switch to support 802.3ad/LACP. Mode 6 bonds with the tagged vlans are supported only up to Oracle VM Release 3.3.1..
  • Virtual Machine(s)
    • The virtual machine network channels provide access to one or more networks provisioned for virtual machines such as public, private, backup and replication networks. 
    • Network Class: A, B or C
    • Server/Network Switch Options: Trunk ports/802.1Q and/or access ports with NIC bonding mode 1, 4 and 6. Mode 1 and 6 do not require special network switch settings. Mode 4 requires the network switch to support 802.3ad/LACP. Mode 6 bonds with the tagged vlans are supported only up to Oracle VM Release 3.3.1..
 
Tip: A best practice is use a minimum of three nodes per cluster to ensure quorum and to be able to generate meaningful network heartbeat (O2CB_IDLE_TIMEOUT_MS timeout) fault tests. A known limitation with two node clusters and network failures causes the node with the higher node number to self-fence.
 

Oracle VM Network Fault Testing

Network Bond Mode, Network Bond failover, and O2CB_IDLE_TIMEOUT_MS Fault Tests:
If a server pool member fails to respond to its network heartbeat, the server pool member is fenced from the pool, promptly reboots, then all HA-enabled virtual machines are restarted on a live node in the pool. The network heartbeat should be placed on a routable or non-routable dedicated class A, B or C network. A best practice is to provision a dedicated network channel for the network heartbeat to avoid network contention and unexpected reboots. The network heartbeat is referred to as the Cluster Heartbeat network channel in Oracle VM Manager.
 
Before an Oracle VM server pool is placed into production, network fault testing should be conducted on the Cluster Heartbeat, Storage and Virtual Machine network channels to find a suitable O2CB_IDLE_TIMEOUT_MS o2cb timeout value, bond mode (1, 4 or 6) and network switch configuration that provides predicable failure response. For example, Oracle VM Servers should be able to lose a bond port/NIC and/or a redundant network switch without node evictions. Incompatible o2cb timeout values, bond modes and network switch configurations can trigger node evictions and unexpected server reboots.
 
Tip: When fault testing a two node cluster's O2CB_IDLE_TIMEOUT_MS time out value, the node with the higher node number will reboot when the network fails. A best practice is use a minimum of three nodes per cluster to ensure quorum and to be able to fault test the O2CB_IDLE_TIMEOUT_MS timeout value.
 
The next tables shows each of the fault test with the expected failure results. The example is four a four port 10G 802.1q/LACP trunk port design with two mode 4 bonds. Modify the table to reflect your design, and your fault tests.
  1. Disable switch ports using the following fault patterns to text NIC, Bond and Switch failures and OCFS2 compatibility.
  2. Use the following commands to confirm the tests results.
  • watch cat /proc/net/bonding/bond0
  • watch cat /proc/net/bonding/bond1
  • tail -f /var/log/messages
Note: While fault testing, run a HA enabled VM on the tested Oracle VM server to confirm that the VM is not interrupted during the single port fault test. When both server management bond ports fail at the same time, the Oracle VM server should fence, the HA enabled VM should restart on a live node.
 
Complete and document each test to confirm which settings provide the expected failure results.
Use Case
Port(s)
Bond Mode
Switch Mode
Expected Results
Results
1- Disable the port and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
 
Enable the port and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
eth0
bond0
 
Mode x
LACP
Bond stays active the node does not self-fence.
 
2- Disable the port and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
 
Enable the port and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
 
eth1
bond0
Mode x

LACP

Bond stays active the node does not self-fence.
 
3- Disable the port and wait 60 secs.
 
Enable the port and wait 60 secs.
eth2
bond1
 
Mode x

LACP

1) Bond stays active the node does not self-fence.
2) VM does not lose connectivity.
 
4- Disable the port and wait 60 secs.
 
Enable the port and wait 60 secs.
 
eth3
bond1
Mode x

LACP

1) Bond stays active the node does not self-fence.
2) VM does not lose connectivity.
 
5- Disable both ports and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
 
Enable both ports and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
eth0
bond0
eth2
bond1
Mode x

LACP

1) Bond stays active the node does not self-fence.
2) VM does not lose connectivity.
 
6- Disable both ports and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
 
Enable both ports and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
eth1
bond0
eth3
bond1
Mode x

LACP

1) Bond stays active the node does not self-fence.
2) VM does not lose connectivity.
 
7- Disable both ports and wait 60 secs.
 
Enable both ports and wait 60 secs.
eth2
bond1
eth3
bond1
Mode x

LACP

Bond loses connectivity, VMs lose connectivity, the node does not self-fence.
 
8- Disable both ports and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
 
Enable both ports and wait +20 secs over the O2CB_IDLE_TIMEOUT_MS time out value.
eth0
bond0
eth1
bond0
Mode x

LACP

1) At the O2CB_IDLE_TIMEOUT_MS timeout value the node self-fences and reboots.
2) After ~60 the VM restartes on a live node
3) The node successfully reboots and joins the pool

 

Oracle VM Storage Fault Testing

If a server pool member fails to read and write storage heartbeats, the server pool member is fenced from the pool, promptly reboots, then all HA-enabled virtual machines are restarted on a live node in the pool. The storage heartbeat should be on a dedicated IP or Fiber channel network. A best practice is to have dedicate “not” shared storage with performance monitoring and storage quotas alerts to avoid contention and full disks that may cause unexpected reboots. A best practice with iSCSI and NFS storage is to provision the storage on a dedicated class A,B or C network channel to avoid network contention and unexpected server reboots.
 
Oracle VM uses two different types of storage repositories. The first type of storage repository, called a pool file system, is used to host a server pool's cluster configurations including the storage heartbeat. There can only be one pool file system per server pool. The other type of storage repository, called a virtual machine file system, is used to host virtual machine configuration files and disks. There can be one or more virtual machine file system repositories in a server pool. Virtual machine file system repositories do not have a storage heartbeat.
 
The storage heartbeat, also known as a “quorum disk”, is used to monitor the status of each Oracle VM Server in a pool. With a quorum disk, every Oracle VM Server in a pool regularly reads and writes a small amount of status data to a reserved section of pool file system. Each Oracle VM Server writes its own status and reads the status of all the other Oracle VM Servers in the pool. If any Oracle VM Server in a pool fails to update its status within its O2CB_HEARTBEAT_THRESHOLD o2cb timeout value, the Oracle VM Server is fenced from the pool, promptly reboots, then all HA-enabled virtual machines are restarted on a live Oracle VM Server in the pool.
 
Before an Oracle VM server pool is placed into production, storage fault testing should be conducted on the quorum disk to find a suitable O2CB_HEARTBEAT_THRESHOLD o2cb timeout value, SAN and FC Switch configurations that provides predicable failure response. For example, Oracle VM Servers should be able to lose one HBA without node evictions. Incompatible o2cb timeout values, SAN and FC Switch configurations can trigger node evictions and unexpected server reboots.
 
Note: While fault testing, run a HA enabled VM on the Oracle VM server that will be tested to confirm that the VM is not interrupted during the single port fault testes. When both HBAs fail at the same time, the Oracle VM server should fence, the HA enabled VM should restart on a live node.
 
The next tables shows each of the fault test with the expected failure results. Modify the table to reflect your design, and yourr fault tests.
 
Use the following commands to confirm the tests results.
  • watch multipath -ll
  • tail -f /var/log/messages

In the following example, each host has two HBAs installed with a single path to each fabric.  Each fabric then has two connections (one to each SP) to the array for a total of four paths on the array-side of the fabric.

Complete and document each test to confirm which settings provide the expected failure results.

Use Case
Port(s)
Bond Mode
Expected Results
Results
1- Disable the port and wait +20 secs over the O2CB_HEARTBEAT_THRESHOLD time out value.
HBA0

ALUA Mode 4

HBA0 paths go down, HBA1 paths stay active, the node does not self-fence.
 
2- Disable the port and wait +20 secs over the O2CB_HEARTBEAT_THRESHOLD time out value.
HBA1

ALUA Mode 4

HBA1 paths go down, HBA0 paths stay active, the node does not self-fence.
 
3- Disable both ports and wait +20 secs over the O2CB_HEARTBEAT_THRESHOLD time out value.
HBA0
HBA1

ALUA Mode 4

After the O2CB_HEARTBEAT_THRESHOLD timeout value the node self-fences.
 

Oracle VM Master Server VIP Failover Testing

Oracle VM facilitates centralized server pool management using an agent-based architecture. The Oracle VM agent is a python application that is installed by default with Oracle VM Server. Oracle VM Manager dispatches commands using XML RPC over a dedicated network using TCP/8899 to each server pool's Master Server. Each Master Server dispatchs commands to subordinate agent servers using TCP/8899. There is only one Master Server agent in a server pool at any one point in time. The Master Server agent is the only server in a server pool to communicate with Oracle VM Manager. Agent intra-component traffic should be isolated to a dedicated class A,B or C Server Management network channel.
 
To address the single point of failure for the Master Server agent, the server pool "Virtual IP" feature was introduced. The Virtual IP feature detects the loss of the Master Server agent and automatically failover the Master Server to the first node that can lock the cluster.
 
To test Virtual IP failover, first confirm which node is the Master Server by accessing Oracle VM Manager => Servers and VMs => Right Click the desired Server Pool => Confirm the host name in the Master Server drop down list. Next, as root access the Master Server, and stop the ovs-agent service to typing “service ovs-agent stop”. After 60 seconds, ssh to the Virtual IP address to confirm that the Master Server agent failed over to a new node.
 
Document Created: 01/03/13
Last Update: 09/15/15
 
Copyright © 2015 Mokum Solutions, Inc. All rights reserved.
Distribution of the Oracle Cloud Cookbook or derivative of the work in any form is prohibited unless prior permission is obtained from the copyright holder.