Make an Admin’s job easier with HACMP/XD-PPRC

HACMP/XD uses PPRC mirroring technology to provide disaster recovery for critical applications' data. To achieve high availability for distributed applications over extended distances, administrators need to configure PPRC in an HACMP/XD environment. This involves collecting configuration data from the storage as well as from the cluster nodes; at times this can be tedious. This article shows you how, using pprc4hacmp, configuration data is generated automatically for HACMP/XD-PPRC.       Introduction The pprc4hcmp script is a tool that automatically generates configuration data for HACMP/XD-PPRC with minimal user input and helps in complex PPRC configuration under the HACMP/XD environment. Users should have minimal High Availability Cluster Multi Processing (HACMP) and IBM® AIX® knowledge to use this tool. A typical PPRC configuration requires identifying the PVIDs,VGnames, and Hostnames, and more. The main intention of the tool discussed in this article is that it generates PPRC-related configuration data that is required to configure HACMP PPRC. Users just need to collect data from this tool and use it in configuring PPRC using SMIT. Those who have base HACMP and AIX knowledge can benefit from this tool by a great deal by leveraging from its centralized automated configuration data discovery. What is HACMP/XD? HACMP Extended Distance (HACMP/XD) is a software solution for disaster recovery, and is an extension to the base HACMP software that enables a cluster to operate over extended distance at two sites. In case of failure at the production site, HACMP/XD moves the application to the backup site, which is at a remote location. When the application is restarted on the remote site, the remote site should contain the data that was used by the application on the primary site, so that the applications can provide services to clients without data loss. As the sites are normally spread across different locations with larger distances, users need to use PPRC or XD technology. HACMP uses different mirroring technologies to achieve this. They are: GLVM -- Geographical Logical Volume Mirroring HAGEO PPRC by IBM System Storage™ DS8000™/DS6000™/ESS800™ SVC-PPRC Customers often choose to use PPRC, as it is faster compared to network XD technologies. This article deals with HACMP/XD using PPRC with IBM System Storage. What is PPRC? Peer-to-peer remote copy (PPRC) is a protocol to mirror a disk from one storage system to another disk in a remote site. PPRC can be used to provide fast data recovery after failure of the primary site. PPRC is classified as synchronous and asynchronous. Synchronous PPRC causes each write to the primary volume to be performed to the secondary as well, and the I/O is only considered complete when the update to both primary and secondary have completed. Asynchronous PPRC I/O is considered to be completed immediately after performing the I/O on primary volume,and performs I/O to the secondary volume asynchronously. The IBM System Storage DS8000, DS6000, and ESS800 devices support PPRC. What is DSCLI? The DS command line interface (DSCLI) provides several commands to configure storage, such as adding host information, assigning disks, and the like. It also provides copy services commands used to configure PPRC. What is ESSCLI? The ESS command line interface (ESSCLI) is similar to DSCLI but is only applicable to ESS800 storage units. But to work with copy services, you can use DSCLI with both DS and ESS. If the HACMP cluster has both DS and ESS as part of the cluster, both DSCLI and ESSCLI need to be installed. Sample PPRC setup A typical PPRC setup is shown in Figure 1. To start PPRC, the minimal required resources are two storage systems connected with FCP/ESCON links. Figure 1. Sample PPRC setup   This sample PPRC setup consists of two storage units; one is located at the production site and another is at a backup site. I have chosen two disks; one is from the DS8000 series with a volume ID of 0620 on the production site, and another is from the ESS800 series with a volume ID 103Fm on the backup site. They are connected with an FCP link. The volume ID of the disk is a combination of the LSS (Logical Sub System) and the disk sequence number. In this case, LSS is 06 and 20 is the sequence number of the disk. The PPRC path is a logical path that is defined between the primary LSS and the secondary LSS. These logical paths are defined over physical links between disk subsystems. Each physical link includes a host adapter in the primary disk subsystem and a host adapter in the secondary disk subsystem. Follow these steps to establish a PPRC relationship: Establish a pprcpath between LSS 06 -> 10 using the mkpprcpath dscli command Establish a pprc relationship between 0620-> 103F using the mkpprc dscli command Check for the pprcstate using the lspprc dscli command. If the state is full duplex, the two disks are in sync. If you are establishing the PPRC relationship, use mode=FULL so that the remote disks will be mirrored completely (including VGDA). If it is already established and you don't want to start mirroring immediately, use copymode=nocopy . In the article, we are dealing with the Metro mirror, so select type=mmir .   Example commands The following code listing is a sample of DSCLI commands. Listing 1. Example DSCLI commands   #/opt/ibm/dscli/dscli -user xxxxx -passwd xxxxx -hmc1 aaa.bbb.ccc.ddd Date/Time: December 17, 2007 1:39:11 AM CST IBM DSCLI Version: 5.2.400.426 DS:IBM.2107-xxxxxxx (To list available PPRC Ports between two storage systems) dscli> lsavailpprcport -l -remotewwnn 500507630EFFFDE4 IBM.2107-7516231/06:IBM.2105-22012/10 Date/Time: March 30, 2007 5:36:35 AM CDT IBM DSCLI Version: 5.2.400.426 Local Port Attached Port Type Switch ID Switch Port =================================================== I0000 I0001 FCP NA NA I0000 I0002 FCP NA NA I0000 I0100 FCP NA NA (To Establish PPRC path between LSS 06 and LSS 10 ) dscli> mkpprcpath -dev IBM.2107-7516231 -remotedev IBM.2105-22012 -remotewwnn 500507630EFFFDE4 -srclss 06 -tgtlss 10 i0000:i0100 Date/Time: April 20, 2007 6:27:42 AM CDT IBM DSCLI Version: 5.2.400.426 DS: IBM.2107-7516231 CMUC00149I mkpprcpath: Remote Mirror and Copy path 06:06 successfully established. (To establish PPRC metro mirror relationship between disks 0620 and 100F ) dscli>mkpprc -dev IBM.2107-7516231 -remotedev IBM.2107-13AAY4A -type mmir -mode full 0620:100F Date/Time: April 20, 2007 6:31:45 AM CDT IBM DSCLI Version: 5.2.400.426 DS: IBM.2107-7516231 CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 0620:100F successfully created.   Configure HACMP/XD and PPRC Say an application is using these disk to store its data and the customer wants to make this application highly available using HACMP. Making application highly available is not sufficient; we should also make the disk data highly available to achieve complete high availability. This is can be achieved by using HACMP/XD with PPRC. Let us consider the example discussed in the Sample PPRC setup section. Figure 2. A simple two-site HACMP/XD cluster with PPRC setup   First, configure a base HACMP/XD cluster which consists of one node per site. Each site has disks assigned from storage. In Figure 2, node1 and DS8000 are the hostname of the server and storage located at siteA, respectively. And node2 and ESS800 are the node name of the server and storage type that are located at siteB. Assuming that we have already established the PPRC relationship, the only thing needed is to integrate this to HACMP. To integrate existing PPRC to HACMP, we have to configure the following: Copy services configuration DS ESS disk subsystems DSCLI Managed PPRC Replicated Resource Let's start with how to collect the required data to configure the items as shown in Figure 2. The disk 0620 is visible as hdisk10 on node1. And the disk 103F is visible as hdisk5 on node2. The user does not know the storage-related information. He just knows the PPRC disks are hdisk10 and hdisk5 and their PVIDs. Note that once the two disks are in PPRC, they acquire the same PVIDs. After establishing the PPRC between the two disks, the remote disks' VGDA is replaced with the VGDA of the local disk so that they have the same PVIDs. Now, if you want to integrate these disks (make VGs on these disks highly available) with HACMP, you need to collect the information like storage ID, volume ID, LSS ID, pri-sec portID, and sec-pri PortIDs. These can be collected using DSCLI commands as mentioned earlier. Instead of going through this process, use the pprc4hacmp script, which generates all this configuration data for you. The assumptions here are: That DSCLI is installed on all nodes and is in the /opt/ibm/dscli directory. You are able to rsh or ssh from the current node to the remote node. Two possibilities are considered: one is if PPRC is already configured (configured here means with respect to storage, not with HACMP) and another is a new PPRC configuration. This article deals with an existing PPRC. You just need to pass the site names, node name (hostnames), PVIDs, and DS/ESS HMC IPS for each storage system. The pprc4hacmp script generates configuration data that you need to enter as part of the HACMP-PPRC configuration. It generates configuration data in two formats: Colon-delimited SMIT snapshot   How to use pprc4hacmp The pprc4hacmp script can be operated in two modes: Interactive mode Non-interactive mode Interactive mode To start in interactive mode, enter the following: #pprc4hacmp   When you run the script as shown above, it prompts for several inputs that are used in generating the required HACMPPPRC configuration. These inputs are needed because in this mode the tool is not using any of the HACMP utilities or commands to get the configured cluster information. Listing 2. Interactive mode example   # pprc4hacmp.sh Enter primary Site Name......: Site A Enter one node from Primary Site ....: Node1 Enter Secondary Site Name ......: SiteB Enter one node from secondary Site ..... Node2 Enter VG name ...: RG1_CG1 Enter PVIDs@SiteA of all disks for the VG -RG1_CG1 separated by space...: 00c46a1d4e588ed3 Enter PVIDs@SiteB of all disks for the VG -RG1_CG1 separated by space...: 00c46a1d4e588ed3 Enter DS HMC ip at SiteA ....: 192.168.1.141 Enter username for 192.168.1.141.....: unamexx Enter password for 192.168.1.141.....: passwordxx Enter Storagetype for 192.168.1.141.....: 2107 Enter DS HMC ip at SiteB ....: 192.168.2.210 Enter username for 192.168.2.210.....: unamexx Enter password for 192.168.2.210.....: passwordxx Enter storage type for 192.168.2.210.....: 2105 validating input........   Non-interactive mode To start in non-interactive mode, enter the following: #pprc4hacmp -i [variablefile]   The variablefile is a file that contains the required input information in a specific manner, as shown in the listing below: Listing 3. Variable file template   PRISITE:sitename:nodename@site1 SECSITE:sitename:nodename@site2 PRIDSINFO:primary HMCip:username:password:storage Type SECDSINFO:primary HMCip:username:password:storage Type VGNAME:volumegroupname which is on pvs PRIPVS:pvids from primary site SECPVS:pvids from secondary site Listing 4. Example variable file   PRISITEINFO:SiteA:Node1 SECSITEINFO:SiteB:Node2 PRIPVs: 00c46a1d4e588ed3 SECPVs: 00c46a1d4e588ed3 PRIDS:192.168.1.141:unamexx:passwordxx:2107 SECDS:192.168.2.210:unamexx:passwordxx:2105 VGNAME:RG1_CG1 Format 1. Non-interactive mode, colon-delimited format   # pprc4hacmp.sh -i varfile -c validating input........ Searching for DSCLI on node Node1 ....... DSCLI found on node Node1 Searching for DSCLI on node Node2 ....... DSCLI found on node Node2 HMC 192.168.1.141 ping test ....... Success Storage Image id managed by HMC- 192.168.1.141 ........... : IBM.2107-7516231 HMC 192.168.2.210 ping test ....... Success Storage Image id managed by HMC- 192.168.2.210 ........... : IBM.2105-22012 Verifying PVs existence at SiteA...... Verifying PVs existence at SiteB..... Verifying PV Vs LSSs on each Site ........ Finding LSS and VOlume PAIRs ...... Finding Pri-sec and sec-pri PortIDs for the Lsss 06 10....... --------------- Copyservices Server Configuration data ------------------- CSSINFO:css_SiteA:DSCLI:SiteA:192.168.1.141:unamexx:passwordxx CSSINFO:css_SiteB:DSCLI:SiteB:192.168.2.210:unamexx:passw0rdxx ----------------------------------****------------------------------------ ------------------ DS ESS Subsystem Configuration data ------------------- DSESSINFO:dsess_SiteA:SiteA:192.168.1.141:unamexx:passwordxx::IBM.2107-7516231 :css_SiteA DSESSINFO:dsess_SiteB:SiteB:192.168.2.210:unamexx:passwordxx::IBM.2105-22012 :css_SiteB ----------------------------------****------------------------------------ -------------DSCLI-Managed PPRC Replicated Resource Configuration--------- PPRC_REP_RES:PPRC_RG1_CG1:SiteA SiteB: 0620->103F:dsess_SiteA dsess_SiteB:06 10: mmir:I0000->I00A4 I0002->I00A4:I0020->I0330 I0020->I0333:FCP:OFF:MANUAL:RG1_CG1 ----------------------------------****------------------------------------   Output can be obtained in two formats, colon-delimited format and SMIT-snapshot format. Format 2. Non-interactive mode, SMIT snapshot format   # pprc4hacmp.sh -i varfile validating input........ Searching for DSCLI on node Node1 ....... DSCLI found on node Node1 Searching for DSCLI on node Node2 ....... DSCLI found on node Node2 HMC 192.168.1.141 ping test ....... Success Storage Image id managed by HMC- 192.168.1.141 ........... : IBM.2107-7516231 HMC 192.168.2.210 ping test ....... Success Storage Image id managed by HMC- 192.168.2.210 ........... : IBM.2105-22012 Verifying PVs existence at SiteA...... Verifying PVs existence at SiteB..... Veryfying PV Vs LSSs on each Site ........ Finding LSS and VOlume PAIRs ...... Finding Pri-sec and sec-pri PortIDs for the Lsss 06 10....... --------------- Copyservices Server Configuration data ------------------- * CSS Subsystem Name [css_SiteA] * CSS Site Name SiteA + * CSS IP Address [192.168.1.141] * CSS User ID [unamexx] * CSS Password [passwordxx] ----------------------------------****------------------------------------ --------------- Copyservices Server Configuration data ------------------- * CSS Subsystem Name [css_SiteB] * CSS Site Name SiteB + * CSS IP Address [192.168.2.210] * CSS User ID [unamexx] * CSS Password [passwordxx] ----------------------------------****------------------------------------ ------------------ DS ESS Subsystem Configuration data ------------------- * ESS Subsystem Name [dsess_SiteA] * ESS Site Name SiteA + ESS Cluster1 IP Address [192.168.1.141] ESS Cluster2 IP Address [ ] * ESS User ID [unamexx] * ESS Password [passwordxx] * Full ESS Storage ID [IBM.2107-7516231] * List of CS Servers [css_SiteA] + ----------------------------------****------------------------------------ ------------------ DS ESS Subsystem Configuration data ------------------- * ESS Subsystem Name [dsess_SiteB] * ESS Site Name SiteB + ESS Cluster1 IP Address [192.168.2.210] ESS Cluster2 IP Address [ ] * ESS User ID [unamexx] * ESS Password [passwordxx] * Full ESS Storage ID [IBM.2105-22012] * List of CS Servers [css_SiteB] + ----------------------------------****------------------------------------ -------------DSCLI-Managed PPRC Replicated Resource Configuration--------- * PPRC Resource Name [PPRC_RG1_CG1] * HACMP Sites [SiteA SiteB] + * PPRC Volume Pairs [ 0620->103F] * ESS Pair [dsess_SiteA dsess_SiteB] + * LSS Pair [06 10] + * PPRC Type mmir + * Pri-Sec Port Pair IDs [I0000->I00A4 I0002->I00A4] * Sec-Pri Port Pair IDs [I0020->I0330 I0020->I0333] * PPRC Link Type FCP + * PPRC Critical Mode OFF + * PPRC Recovery Action MANUAL + * Volume Group [RG1_CG1] ----------------------------------****------------------------------------   Configuring CSS, DSESS, and PPRCs The user uses the configuration data and smit pprc_ds to configuration HACMP. Listing 5. Where to use the generated configuration data   smit pprc_ds -------------------------------------------------------------------- Copy Services Server Configuration DS ESS Disk Subsystem Configuration DSCLI-Managed PPRC Replicated Resource Configuration PPRC Consistency Groups Configuration Verify PPRC Configuration   From the above smit menu: Choose the Copy Services Server Configuration option and configure CSS Choose DS ESS Disk Subsystem Configuration and configure DS ESS subsystem Choose DSCLI-Managed PPRC Replicated Resource Configuration and configure the PPRC-replicated resource   Summary You only need basic knowledge of HACMP and AIX to use this utility. Using the configuration data generated by the pprc4hacmp tool can save you time as well as reduce or eliminate the chances of mistake in configuration. The user does not need to know about DSCLI commands, format of output, and different fields of output. So users can use this script to configure PPRC easily.