DB2 and the Live Partition Mobility feature of PowerVM on IBM System p using storage area network (SAN) storage

Learn about Live Partition Mobility, a feature of the System p™ virtualization PowerVM™ Enterprise edition. See how Live Partition Mobility can be applied to DB2® deployments, and how it helps you migrate AIX® and Linux® partitions and hosted applications from one physical server to another compatible physical server. Live Partition Mobility allows hardware maintenance, firmware upgrades, system maintenance, and on-the-fly server consolidation without application outage. Setup, configuration, best-practices, and performance characterization for Storage Area Network (SAN) and DB2 are covered.       Introduction System p virtualization technologies offer a rich set of platform deployment features. Features range from simple resource isolation to an array of the most advanced and powerful functions, including: server-resource partitioning, autonomic dynamic resource reallocation, and workload management. Live Partition Mobility, a new feature on POWER6™ processor-based systems, is designed to enable migration of a logical partition (LPAR, or partition) from one system to another compatible system. Live Partition Mobility uses a simple and automated procedure that transfers the migrating LPAR’s running state (primarily memory pages belonging to the migrating LPAR) and configuration information (such as virtual network interface, virtual Small Computer System Interface (SCSI) interface, or LPAR profiles) from the source server to the destination server without disrupting the hosted operating system, network connectivity, and hosted applications. Live Partition Mobility gives the administrator greater control over the system resources in a data center. It increases flexibility in workload deployment and workload management, which previously wasn't possible due to narrow maintenance windows, stringent application availability requirements, or strict service level agreements that don't allow an application to be stopped. The migration process can be performed either with a partition in powered off state or a live partition. There are two available migration types: Inactive migration The logical partition is powered off and moved to the destination system. Active migration The migration of the partition is performed while service is provided, without disrupting user activities.   Maintenance activities that previously required down time can now be performed with no disruption to services. Activities include, but are not limited to: Preventive hardware maintenance Hardware upgrades Server consolidation that involves reallocation of partitions among servers   DB2 V9.5 is the industry’s leading information management platform. DB2 V9.5 is designed to leverage the System p virtualization technology. Innovative features such as host dynamic-resource awareness, online tuning of configuration parameters, and self-tuning memory management (STMM) make DB2 well suited for the virtualization environment.     Prerequisites for Live Partition Mobility It is assumed that you are knowledgeable about System p virtualization concepts, storage management, network administration, and basic system administration skills. Live Partition Mobility has specific requirements in terms of the operating system level, firmware level, DB2 data storage layout, and network interfaces. Setting up for the migration, regardless of whether it's active or inactive, requires careful deployment planning beforehand. Sometimes you can qualify a partition for mobility by taking additional steps, such as removing physical adapters (non-virtual adapters) using a dynamic logical partitioning (DLPAR) operation. Only the partitions meeting the following requirements are eligible for migration. The typical requirements for the migration of a logical partition are: Two POWER6 processor-based systems controlled by the same Hardware Management Console (HMC). An optional redundant HMC configuration is supported.   The destination system must have at least an equal amount of available memory and processors as the migrating partition’s current usage. DLPAR operation might have altered partition resources (including memory, processor, and adapters) at run time after it is booted with a partition profile. Hence, the current amount of partition resources may be different than the amount specified as the “desired” in the partition profile. A partition can have multiple profiles. At operating system boot time, the system administrator selects which profile a partition is booted with. In general, a partition profile specifies operating system boot time resource requirements, such as the desired amount of memory, processor entitlement, adapters, and so on. The partition profile also specifies the minimum and maximum amount of memory, and number of virtual processors and processor entitlement. The operating system, applications (DB2) installation, DB2 instance data, and DB2 table data must reside on Virtual I/O storage on an external storage subsystem.   The mobile partition's all network and disk access must be virtualized using one or more Virtual I/O Servers (VIOS).   No other physical adapters, such as serial I/O adapter, SCSI adapters, and so on, may be used by the mobile partition at the time it is migrated. The pre-migration validation phase will fail if any physical adapter belongs to a migrating partition.   The VIOS on both systems must have a shared Ethernet adapter configured to bridge to the same Ethernet network used by the mobile partition.   The VIOS on both systems must be capable of providing virtual access to all storage resources that the mobile partition is using. Figure 1 shows a general schematic to enable mobility. Each POWER6-based system is configured with a VIOS partition. The mobile partition has access to network and disk resources only through virtual adapters. The VIOS on the destination system is connected to the same network, and is configured to access the same storage used by the mobile partition. Figure 1. Partition mobility infrastructure     Configuration to enable Live Partition Mobility This section provides a reference checklist of configuration settings for Live Partition Mobility. In general, one method to verify items in the checklist is to navigate to the respective Property screen. For example, for items in the Systems checklist navigate to System -> Property screen. For a VIOS partition, navigate to its Property screen. The Redbook PowerVM Live Partition Mobility on IBM System p (see Resources) has detailed instructions for verifying and changing these settings. Systems The source and destination servers must be at least POWER6 processor architecture-based systems. Both systems must be controlled by the same HMC. Live Partition Mobility requires HMC Version 7.3.2 or higher. The memory region size (also called logical memory block (LMB) size) must be the same on the source and destination systems. It can be verified on HMC by navigating to the Memory tab on the managed system Properties screen. See Figure 2 below. The destination system cannot be running on battery power for migration to take place. The destination system must have at least an equal amount of processors and memory available (as dictated by the active partition profile) as the host mobile partition. Figure 2. Verify system memory region (LMB) size on HMC   VIOS partitions At least one VIOS at release level 1.5 or higher has to be installed both on the source and destination systems. The partition attribute “Mover service partition” (MSP) must be enabled on both the source system and target system VIOS. You can enable this by selecting a check box when creating the VIOS partition profile, or later by navigating to the partition property and selecting a check box, as shown in Figure 3. On the VIOS where MSP is enabled, a Virtual Asynchronous Service Interface (VASI) device must be defined and configured in order to be considered a valid MSP candidate. Virtual disks for use by the mobile partition have to be backed by the devices; they cannot be part of any logical volume group on VIOS. Figure 3. Enable or check status of mover service partition   Mobile partition AIX 5.3 technology level 7 or later. Ensure that Resource Monitoring and Control (RMC) connections are established. Redundant error path reporting has to be disabled on the migrating partition. The mobile partition cannot have any required virtual serial adapters, except for the two reserved for the HMC. The mobile partition cannot be part of a partition workload group. A partition workload group is a group of logical partitions whose resources are managed collectively by a workload management application. Workload management applications can balance memory and processor resources within groups of logical partitions without intervention from the HMC. The mobile partition cannot use barrier synchronization register (BSR) arrays. The mobile partition cannot use huge (16GB) memory pages. The 64KB and 16MB AIX page sizes are allowed, however. The mobile partition does not have physical or dedicated I/O adapters and devices. The mobile partition cannot be configured with any logical host Ethernet adapter (LHEA) devices. A host Ethernet adapter (HEA) is a physical Ethernet adapter that is integrated directly into the system (GX+ bus) on a managed system. HEAs offer high throughput, low latency, and virtualization support for Ethernet connections. HEAs have the same uses as other types of physical Ethernet adapters. For example, you can use an HEA to establish a console connection to a logical partition. External storage The same Storage Area Network (SAN) disks used as virtual disks by the mobile partition must be assigned to the source and destination Virtual I/O logical partitions. reserve_policy attributes on the shared physical volumes must be set to no_reserve on the VIOS. The physical volumes must have the same unique identifier, physical identifier, or an IEEE volume attribute. This can be verified using the AIX lsattr command. The VIOS must be able to accurately identify a physical volume each time it boots, even if an event such as a SAN reconfiguration or adapter change has taken place. Physical volume attributes, such as the name, address, and location, might change after the system reboots due to SAN reconfiguration. However, the VIOS must be able to recognize that this is the same device and update the virtual device mappings. To export a physical volume as a virtual device, the physical volume must have either a unique identifier (UDID), a physical identifier (PVID), or an IEEE volume attribute. The mobile partition must have access to the storage through the VIOS. The destination VIOS must have sufficient free virtual slots to create the virtual SCSI adapters required to host the mobile partition. The mobile partition must have access to the same physical storage on the SAN from both the source and destination environments. Network considerations A shared Ethernet adapter has to be configured on both the source and destination VIOS. A virtual Ethernet adapter has to be configured on the mobile partition.     Mobility in action From the end user’s perspective, an active migration is simply a few clicks on the HMC graphical user interface. Behind the scenes, a lot of work is involved in transferring the migrating LPAR state and configuration information from the source to the destination system (including many gigabytes of memory configured in the migrating LPAR), while keeping all services of the LPAR functional, and maintaining coherency between the source and destination systems. An active migration involves: A validation phase to make sure migration criteria are met. The validation process essentially verifies the prerequisites listed above. New LPAR creation on the destination system. MSP setup on both VIO servers. Virtual SCSI adapter setup on the destination VIO server. Memory copy from the source to the destination system. Suspend the LPAR on the source system and resume it on the destination system. Virtual SCSI removal from the source VIO server. Notification of completion to the mobile partition and VIO servers. Removal of the LPAR from the source system.     Test plan The test plan outlines the test environment and test cases. Test environment The following table summarizes the system, storage, and software used for the proof of concept. Table 1. Environment   Hardware System: Two systems, each System p 570 with 4x 4.7 GHz POWER6 cores with IBM PowerVM Enterprise edition feature Number of cores used: Four Physical memory: 9 GB Firmware version: EM320_031 HMC version: V7R3.2.0.0 Disk characteristics: Number of disks: 28 RAID5 FCP LUNs. 196 external disks (140 data + 56 transaction logs) Type, size, speed: FCP, 18 GB, 15 000 RPM Software Operating system: AIX 5L v5.3 TL07, 64-bit kernel Database: DB2 9.5 for AIX, 64-bit Workload Characteristics: Online-transaction processing (OLTP) Database size: ~100 GB   Test cases The test cases are designed to understand the following characteristics of the DB2 performance under active migration. The DB2 server instance continues to serve clients at the time of migration. Performance impact at various phases of the migration. Observe client or end user perceived throughput during active migration. How soon the original throughput is reestablished on the destination server. Effect of load on the migration time.       Setup for mobility Figure 4 shows the setup used for the DB2 and Live Partition Mobility performance characterization. Throughout the test, the setup evolved around best practices to ensure ease in management and higher performance throughput. As mentioned, preparation for mobility requires careful storage and network planning. Figure 4. DB2 live partition mobility hardware setup   Hardware setup This section describes the system, storage, and network setup required to enable the active migration. System #1 in Figure 4, above, is considered the source system. The LPAR (running a DB2 server instance) was originally defined on this system. System #2 was set up to receive a migrating DB2 server LPAR. The DB2 server LPAR does not have any physical resource (including physical adapter, or local disk) belonging to it. This is one of the major requirements for mobility to work. The required network and storage interfaces are a shared Ethernet adapter (SEA), and a virtual SCSI, respectively. The DB2 network client is connected to the DB2 server by an SEA, whereas DB2 server data is hosted off the DS4500 using a virtual SCSI adapter. The storage system is hooked up using a SAN switch to be able to connect to both the systems. The important step in configuring SAN storage is to ensure that worldwide port numbers (WWPN) are appropriately declared so both systems can access data. (At any point in time, though, only one system is accessing the storage system.) Configuration to enable Live Partition Mobility has detailed setup and configuration steps. PowerVM Enterprise edition, a separately licensed, paid feature, must be enabled on both the systems to enable partition mobility. The minimum required AIX operating system is version 5.3 technology level 07. Specific minimum HMC and firmware versions are also required. Complete details about the environment are in Test environment . It is mandatory that you install the AIX operating system on virtual storage backed by the SAN storage device backing. It is required so that the AIX install media (specifically, rootvg) will also move along with the migrating partition. To minimize migration latency time, high speed network infrastructure such as Gigabit or 10 Gigabit interfaces are recommended at both ends. To improve network throughput, if possible connect both the systems and HMC with the same high speed Ethernet switch, as shown in Figure 4. DB2 setup Just as you need an AIX installation on SAN storage, your DB2 installation location, DB2 instance directory, DB2 catalog, and all other storage (including DB2 tablespaces containers) also must be on SAN storage. The DB2 database cannot use local storage devices. Be sure that the DB2 instance is not using any physical network adapters, disk adapters, or devices. The migration is not possible with physical adapters belonging to a migrating LPAR. The physical adapters can be removed using the DLPAR operation before the migration. If DB2 is accessing such devices, then the DLPAR operation to remove them will fail, resulting in a migration validation failure. The best way to avoid this situation is to follow the rule that there should be no physical adapters belonging to the LPAR. All adapters (network, storage) must be virtual adapters. PowerVM Virtualization on IBM System p Introduction and Configuration (see Resources) has details about System p virtualization setup and configuration.   Partition mobility is not supported if the LPAR is using huge AIX pages (16GB page size). It's important to ensure that DB2 is not using AIX with huge pages configuration; for example, using DB2 registry variable DB2_LARGE_PAGE_MEM=DB:16GB. The AIX 4KB, 64KB, and 16MB page sizes, which DB2 supports, are supported for use with the partition mobility. The DB2 database manager configuration parameter INSTANCE_MEMORY controls the amount of memory used by the DB2 instance. If the purpose of the migration is to assign more memory (by performing a DLPAR operation after the migration to add more memory to the LPAR hosting DB2) for the DB2 instance on the target system, the INSTANCE_MEMORY configuration parameter may require attention after the migration to be able to use additional memory on the target system. There is no need for any other change in the DB2 configuration.     Migration analysis This section discusses migration performance, LPAR suspend/resume duration, effect of workload intensity, and resource requirements. Active migration performance profile The shaded area in Chart 1 below shows when the mobile LPAR is migrating. Execution of the mobile LPAR on the source system ends with a suspension (in green). It's followed by a resume operation (the red shaded area) that is observed between suspension on the source system and resumption of the target system. Chart 1. Performance profile of DB2 active migration   Notice that the memory copy goes beyond the point of resumption of execution of the mobile LPAR on the destination system. When the LPAR is suspended, some memory pages that were recently modified have not been copied over to the destination system. When processes access memory pages that are still on the source system, the memory pages are demand-paged over from the source system. Therefore, processes can continue execution on the destination without waiting for all the remaining memory pages to be copied over. Chart 1 also shows that database transactions continue once the mobile LPAR is resumed, and they return to full performance within a few seconds. This timing is workload dependent. Table 2 below summarizes elapsed time at various stages of the active migration. The times are for example purposes only, and not representative. Times depend entirely on the workload, workload type, DB2 memory consumption, network adapter speed, network topology, amount of memory, storage system, and so forth. Table 2. DB2 live partition mobility summary   Event Duration (mm:ss) Pre-migration validation and state transfer (memory copy) 03:15 Total migration time (elapsed time between the “Begin migration” and “End migration” markers) 03:24   The whole migration scenario took only three minutes and 24 seconds. This time is a function of the amount of memory in an LPAR, and how frequently memory pages are updated during the migration. Further breakdown of the total migration time can be divided into two major events: state transfer took 3 minutes and 15 seconds, followed by around 9 seconds of suspend/resume duration. The original throughput (the throughput at or before the “Begin migration” marker) is established almost instantly on the target system. As seen in Chart 1, transaction throughput is affected during the state transfer phase. The average throughput dropped by about 12% between the markers “Begin migration” and “Suspend on source system.” It is worth reemphasizing that all the performance analysis metrics and elapsed times mentioned in this discussion are examples only. They will vary widely from environment to environment. The next section discusses test results with different levels of load on the mobile LPAR in combination with different VIOS configurations. Load scenarios The next set of tests examines the effect of LPAR load on the mobility characteristics. There are three load levels synthesized by a varying number of DB2 clients. The resulting average DB2 transaction throughput is approximately 1500, 1800, and 2200 transactions/second (TPS), as shown in Chart 2. Chart 2. Effect of partition load on mobility profile   There were not any other changes in the LPAR, AIX, or DB2 configuration. It is clear that the amount of load has apparent effect on the mobility metrics, including total migration time, state transfer time, and suspend/resume duration. The detailed analysis of exact metrics is shown in Table 3. Table 3. Effect of workload intensity on live partition mobility   Event Low Medium High Pre-migration validation and state transfer (memory copy) 02:16 02:33 03:15 Suspend/resume duration 00:09 00:09 00:09 Total migration time (elapsed time between the “Begin migration” and “End migration” markers) 02:25 02:42 03:24   Processor consumption This section reviews processor consumption behavior during the various stages of active migration operation. Chart 3 below represents the same run scenario as in Chart 1, but this time showing processor utilization for each of the involved partitions. Each VIOS (on the source and target system) are configured as a dedicated LPAR, with one processor, having 512MB of physical memory. The DB2 LPAR is configured as a dedicated LPAR with two processors, having 9GB of physical memory. Chart 3. Processor utilization of VIOS and DB2 LPAR during active migration operation   Before the “Begin migration” event, DB2 LPAR processor utilization is around 85%. Right after the migration is triggered, DB2 LPAR processor utilization slightly increases to around 87%, and stays there until the “End migration” marker. The slight DB2 partition processor utilization increase is attributed to the DB2 partition participating in the state transfer. Compared it with Chart 1, where at the same time DB2 throughput starts gradually declining up to the time marker 120 in the charts. After the marker 120, DB2 throughput stays at the same level (down by around 12% as compared to pre-migration level). Before the migration, the source VIOS is servicing DB2 disk and network I/O requests, so it has processor utilization of around 18%. Since there's no other activity on the target system, the VIOS processor utilization is 0%. After the 120 marker in the chart, both VIOS processor utilization increases significantly because now both VIOS are conducting the state transfer operation. The source VIOS still continues servicing the DB2 disk and network I/O, in addition to performing the state transfer operation, causing processor utilization to jump to as high as 90%. Around the same time, the target VIOS processor increases from 0% to around 60% and stays at that level until the end of the state transfer operation. Right before the suspend/resume operation, the target system VIOS processor utilization increases to as high as 95%, the source VIOS shows a similar trend, and so does the DB2 LPAR. Observe that after the “End migration” event, the VIOS processor utilizations are reversed. The target system VIOS processor utilization is at the same level as the source system VIOS processor utilization, since now the target system is servicing the DB2 client, and vice versa. To minimize total active migration duration, it is essential to size the VIOS appropriately. It is recommended that you use a partition type of shared dedicated capacity, or an uncapped shared-processor VIOS partition. To derive the initial amount of processing capacity, set the VIOS partition type as an uncapped shared-processor partition. Do a trial run with a peak workload, perform the active migration, and record the VIOS entitled capacity utilization (using the VIOS viostat command, for example). As a safeguard, add 10% spare capacity to the recorded peak entitled capacity utilization. Round up this value in the case of the shared dedicated partition. With an uncapped shared-processor partition type for VIOS, set the VIOS uncapped weight to the highest value, with entitled capacity calculated in the above step.     Summary Live Partition Mobility gives administrators greater control over the usage of resources in your data center. The migration process can be performed either with a partition in powered off state, or with a live partition. There are two available migration types. With inactive migration, the logical partition is powered off and moved to the destination system. With active migration, the migration of the partition is performed while service is provided, without disrupting user activities. Maintenance activities that required down-time in the past can now be performed with no disruption to services. These activities include, but are not limited to, preventive hardware maintenance, hardware upgrades, and server consolidation that involves reallocation of partitions among servers. The separately licensed product PowerVM Enterprise edition is required to enable the mobility. The PowerVM Enterprise edition includes Live Partition Mobility, Shared Dedicated Capacity, Micro-partitioning, VIOS, and more. This article provided detailed analysis for DB2 9.5 running with the PowerVM Enterprise edition Live Partition Mobility feature. The DB2 9.5 instance is hosting an OLTP workload. The DB2 instance is servicing a multitude of clients, as more than 2000 transaction/second (TPS) throughput is migrated to another system. The client network connections remained intact, and the application observed a blip for only a few seconds. Acknowledgements The authors would like to thank Horace Tong for allowing them to derive some of the content for this article from his previous work; Sunil Kamath and Peter Kokosielis for their help with DB2 setup and configuration; Rich Bassemir for review comments; and Pete Jordan for assistance with lab support and administrative matters related to the system and storage used.