Complex networking using Linux on Power blades

Blades are an excellent choice for many applications and services, especially in the telecommunications service provider industry. But the unique requirements of these provider networks often require configurations that are complex and need up-front focus and planning so all the stringent functional requirements are met. In this article, learn how to plan and set up the necessary network configurations for a POWER6™ JS22 blade deployment.       Blade-based operational models are of tremendous value in the wired and wireless telecommunications domain for several reasons: Small footprints mean cost-effective use of data center space. Deployment meets NEBS requirements for distributed deployments. (NEBS, or Network Equipment Building System, is a set of criteria that networked equipment must meet to qualify for compatibility.) Cost-effective massive horizontal scalability lowers deployment costs for the telecommunications service provider. Centralized management support provides better OAM&P support for in-network deployment in service provider networks. (OAM&P stands for "operations, administration, maintenance, and provisioning." The term describes this collection of disciplines and specific software used to track it.) Built-in support for continuous-availability-based operational models—including upgrades and maintenance activities—avoids service downtime from a subscriber perspective. These additional considerations are key in a telecommunications service provider environment, especially one with complex configurations: Multiple VLANs. These are used for CDN (Customer Data Network) and Management (OAM&P) traffic. Considering them separately ensures that subscriber QoS (Quality of Service) is effectively maintained across multiple LPARs (logical partitions). Micro-partitioning and virtualization. These strategies help maximize the capacity utilization and TCO (Total Cost Of Ownership). Existing network complexities. Existing networks can have a higher degree of load variability, requiring resource load balancing among multiple client LPARs. In this article, we describe an implementation of a multiple VLAN configuration of a blade chassis using Cisco switches paired in an active/passive configuration. In our example, we configured networks to connect multiple VLANs on a BladeCenter® JS22 using Linux® on Power. The architecture consists of six Cisco catalyst switch modules with fourteen internal ports and four external 1GB ports each. To properly leverage all six switches in the chassis, the blade requires six Ethernet interfaces. Ethernet interface ent0 on the blade maps to the first switch on the blade chassis; each consecutive Ethernet interface maps to the next available switch. This mapping creates a limitation, because it does not allow administrators to map the physical adapter on the blade to the switch they choose to map them to. When creating network architecture for your blades, you must have one physical interface for each Cisco switch you want to use on the chassis. If some blades do not have the same number of adapters as the chassis, the switches that do not have a physical adapter associated with them on the blade cannot be used by the blade. It is important to understand how to pair the Ethernet interfaces on the blade and the switches within the chassis. The first switch in the chassis is normally located in the upper left of the blade chassis, just below the power plug. It would map to ent0 on the blade since it is the first interface on the blade. Figure 1 shows the numbering of the switches in our configuration. Figure 1. Physical adapter switches in the configuring we're using   Determining switch pairing is extremely important for high availability. In a typical configuration, one power distribution unit (PDU) supplies power to the top half of the blade chassis while another PDU provides power to the bottom half of the chassis. If you are creating a redundant solution, it's important to split the primary and secondary switch between the upper half and lower half of the chassis. In our case, we created a solution with the following switch pairs: (1, 9) (2, 3) (4, 7). Because adapter pairs (ent0, ent1), (ent2, ent3), and (ent4, ent5) are on the same physical I/O card, we also needed to make sure that the network traffic for our target VLANs did not travel across the same I/O card. Our configuration splits traffic across multiple PDUs and across multiple interfaces. Even though the pairing of the adapters and switches may seem simple, there are multiple steps to configure the IVM (Integrated Virtualization Manager), switches, and LPARs to take advantage of this architecture. Figure 2 represents the setup of one of our blades with the associated switch, trunking, and VLAN tagging. This configuration allows multiple VLANs to communicate over the same physical adapters to multiple switches. Figure 2. One of the blades with the associated switch, trunking, and VLAN tagging   In this example, each LPAR has two Ethernet adapters that connect to one Virtual Ethernet Adapter (VEA) on the IVM. Notice that the VEA on the IVM has multiple VLANs associated with it. Traffic for each VLAN is separated on the LPARs by their adapters. The VEA trunks the VLANs together and sends them over the Shared Ethernet Adapter (SEA) through a Link Aggregation Device and out to the network via one of the chassis switches. The switches route the VLAN traffic to the appropriate network through the use of VLAN tagging. Five steps to configuring VLANs There are five main steps (and one optional step) for configuring VLANs for client LPARs on the IVM: Configure Cisco switches. (An optional step at this point may be to create a link aggregation device, commonly referred to as the LNAGG device.) Create 802.1q IEEE VLAN-aware interfaces on the IVM to support the VLANs you want to use on each interface. It is very important to design the VLANs prior to any work because it is not possible to modify the created VLAN-aware interfaces afterwards. You would need to delete them and re-add them, which means a big waste of time. Assign the Virtual Ethernet Adapter to the physical adapter (LNAGG) in the Virtual Ethernet Menu on the IVM. Modify the LPARs properties to map the new virtual adapters to LPAR adapters. Make sure the LPARs are inactive before performing changes to the network device properties. Boot each LPAR and configure the new interfaces. The example in this article is intended for a fresh install of a blade.     Step 1: Configure the Cisco switches You may choose to skip this step if the switches are already properly configured with the VLANs that you want to use. The example does not demonstrate how to configure spanning trees and switch priority for spanning trees that may be required. If you want to follow the example, then you should configure your switches to match the configuration that follows. Log in to the switch Type the following commands in this order: enable config interface GigabitEthernet0/1 description blade1 switchport trunk native vlan 383 switchport trunk allowed vlan 373,383 switchport mode trunk The commands configure port 1 on the switch for trunking. If traffic comes in from the blade to the switch or from the external ports of the blades to the switch and they are not tagged with a VLAN, the switch will tag them with VLAN 383. The switchport (external ports) will allow only traffic from 373 and 383, in this case, to route through to and from the IVM's VEAs. To change which VLANs need to access the port from the IVM, simply change the trunk-allowed VLANs numbers. Splitting the traffic Once the traffic is trunked and sent to the switch, the switch will determine which external port routes the VLAN traffic. In the example, we are sending external VLAN 373 traffic across port 17, and VLAN 383 over port 20. To configure VLAN 373 traffic over port 17, type the following commands on the Cisco switch: interface GigabitEthernet0/17 description extern1 switchport access vlan 373 switchport mode access To configure VLAN 383 traffic over port 20, type the following commands on the Cisco switch: interface GigabitEthernet0/20 description extern4 switchport access vlan 383 switchport mode access After setting the external port configuration, type exit twice to back out of the configuration mode of the command line. Then run the show run command to see your configuration. The show run command displays the actively running switch configuration. The configuration has not been written to memory in the switch, but you can see the changes that are currently running on the switch. If you look in the configuration, you can see changes we made in the steps above. Look for the Ethernet port for blade 1 as we configured it: Listing 1. Displaying the configuration of switch port 1   interface GigabitEthernet0/1 description blade1 switchport trunk native vlan 383 switchport trunk allowed vlan 373,383 switchport mode trunk   If you issue a show config command, you will see the previous configuration of the interface, not the one we just configured. If the switch is rebooted in this state, the current configuration will be lost. To write it to memory, type write on the command line of the switch. If you run a show config again, the configuration stored on the switch for our interfaces will match the running configuration.     Step optional: Create a Link Aggregation Adapter A link aggregation device can be used to connect two physical adapters together to look like one adapter. This is useful for creating an active/passive configuration for failover. In our example, we wanted to create an active/passive configuration on the blade linking adapter ent0 and ent5 together. Issue the following command to create link aggregation device (LNAGG) with a backup adapter on ent5 from the IVM's command line: $ mkvdev -lnagg ent0 -attr backup_adapter=ent5 Use the lsdev command to verify the creation of the LNAGG device. Listing 2. Verifying the creation of the LNAGG device   $lsdev |grep ^ent ent0 Available Logical Host Ethernet Port (lp-hea) ent1 Available Logical Host Ethernet Port (lp-hea) ent2 Available Gigabit Ethernet-SX Adapter (e414a816) ent3 Available Gigabit Ethernet-SX Adapter (e414a816) ent4 Available Gigabit Ethernet-SX PCI-X Adapter (14106703) ent5 Available Gigabit Ethernet-SX PCI-X Adapter (14106703) ent6 Available Virtual I/O Ethernet Adapter (l-lan) ent7 Available Virtual I/O Ethernet Adapter (l-lan) ent8 Available Virtual I/O Ethernet Adapter (l-lan) ent9 Available Virtual I/O Ethernet Adapter (l-lan) ent10 Available EtherChannel/IEEE 802.3ad Link Aggregation ent11 Available EtherChannel/IEEE 802.3ad Link Aggregation ent12 Available EtherChannel/IEEE 802.3ad Link Aggregation ent13 Available Virtual I/O Ethernet Adapter (l-lan) ent14 Available Virtual I/O Ethernet Adapter (l-lan) ent15 Available Virtual I/O Ethernet Adapter (l-lan) ent16 Available Shared Ethernet Adapter ent17 Available Shared Ethernet Adapter ent18 Available Shared Ethernet Adapter   In the output from our lsdev command, you can see that the link aggregation device (ent10 - ent12) looks like a physical adapter. This allows you to map the link aggregation device to a virtual device (ent13 - ent15) via a Shared Ethernet Adapter (ent16 - ent18). The SEA treats the link aggregation device (LNAGG) as though it is a physical adapter. The Virtual Ethernet Adapters (VEA) ent6 - ent9 are created by default and are not VLAN-aware devices, nor can you modify them to become VLAN aware. ent0 through ent5 are the physical adapters on the blade server. Use the attr flag on the lsdev command to see how the LNAGG device maps to its physical adapters: $lsdev -dev ent10 -attr will produce mapping as shown in Table 1. Table 1. LNAGG to PA mapping with $lsdev -dev ent10 -attr   Attribute Value Description Is it user-settable? adapter_names ent0 EtherChannel Adapters True < Primary alt_addr 0x000000000000 Alternate EtherChannel Addr True auto_recovery no Enable automatic recovery after failover True backup_adapter ent5 Adapter used when whole channel fails True < Backup hash_mode default Determines how outgoing adapter is chosen True mode standard EtherChannel mode of operation True netaddr   Address to ping True noloss_failover yes Enable lossless failover after ping failure True num_retries 8 Times to retry ping before failing True retry_time 1 Wait time (in seconds) between pings True use_alt_addr no Enable Alternate EtherChannel Address True use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True     Step 2: Create virtual adapters on the IVM In this example, we're going to configure network traffic to flow through the Virtual Ethernet Adapter for VLANs 373 and 383. The first step in configuring the solution is to create the virtual adapter on the IVM that is needed to transport traffic to and from the client LPARs. Let's create a virtual adapter with a primary port of 373 and a secondary port of 383. To create the VLAN-aware interface on the IVM: Use your favorite (putty) telnet program, open a window to the IVM, and log in as padmin (the default password is "passw0rd" with a "0" instead of an "O"). Use the shwres -r virtualio --rsubtype eth --level lpar command to list the Ethernet resources: Listing 3. Listing the Ethernet resources   $lshwres -r virtualio --rsubtype eth --level lpar lpar_name=IVM_01,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B503 lpar_name=IVM_01,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B504 lpar_name=IVM_01,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B505 lpar_name=IVM_01,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B506   Use the chhwres command to create a virtual adapter on the IVM that supports IEEE VLAN awareness and the additional VLANs you want on that interface. To do this, issue the following command from the command line of the IVM: $ chhwres -p IVM_01 -o a -r virtualio --rsubtype eth -s 15 -a\ '"ieee_virtual_eth=1","port_vlan_id=373","addl_vlan_ids=383","is_trunk=1","trunk_priority=1"'. The chhwres command tells the IVM how to construct a new VLAN-aware Virtual Ethernet Adapter. There are some important features of the command that you need to know to create multiple virtual adapters on the IVM: -p partition: In the command, we are telling the chhwres that there is a change to the IVM partition by issuing the -p command. -s nn: This tells the IVM that we are going to use a particular slot number. If this parameter is not specified, the IVM will use the next available slot. The slot number is required when a device is removed from the IVM. ieee_virtual_eth: A value of 1 informs the IVM that this adapter supports IEEE 802.1Q. This needs to be set to 1 if there are additional VLANs that are required. port_vlan_id: This is the primary VLAN for the virtual adapter. add_vlan_ids: If trunking is enabled, then this parameter accepts the additional VLANs. is_trunk: This attribute must also be set to 1 if you are passing multiple VLANs. trunk_priority: When trunking, the priority of the adapter must be set between 1-15. Ensure the creation is complete by re-running the lshwres command and look for the new devices. Listing 4. Displaying the new devices   $lshwres -r virtualio --rsubtype eth --level lpar lpar_name=IVM_01,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B503 lpar_name=IVM_01,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B504 lpar_name=IVM_01,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B505 lpar_name=IVM_01,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4, addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B506 lpar_name=IVM_01,lpar_id=1,slot_num=15,state=1,ieee_virtual_eth=1,port_vlan_id=383, addl_vlan_ids=378,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=463337C4B50F lpar_name=IVM_01,lpar_id=1,slot_num=16,state=1,ieee_virtual_eth=1,port_vlan_id=6, "addl_vlan_ids=22,23",is_trunk=1,trunk_priority=1,is_required=0, mac_addr=463337C4B510 lpar_name=IVM_01,lpar_id=1,slot_num=17,state=1,ieee_virtual_eth=1,port_vlan_id=7, "addl_vlan_ids=565,566,567,568",is_trunk=1,trunk_priority=1,is_required=0, mac_addr=463337C4B511       Step 3: Assign the virtual Ethernet ID to a physical adapter Once the link aggregation device has been created, it needs to be mapped to a virtual adapter. This is easily accomplished via the IVM GUI as shown in Figure 3. Figure 3. Mapping the LNAGG to the virtual adapter via the IVM GUI   After logging into the GUI, select the "View/Modify Virtual Ethernet" from the left-side navigation, then choose the "Virtual Ethernet Bridge" tab. From this menu you can see the virtual Ethernet adapter we created previously with the 383 primary VLAN and the 373 additional VLAN. From the drop-down box, you can select the link aggregation device we created in the previous step. Once the new device has been chosen, click Apply. This creates a Shared Ethernet Adapter (SEA) within the IVM.     Step 4: Modify the LPAR's properties Once a physical adapter or LNAGG device has been mapped to a Virtual Ethernet ID on the IVM, then the virtual adapter needs to be created for each logical partition. The first step is to log into the GUI on the IVM (see Figure 4). Figure 4. Creating the virtual adapter for each LPAR   After logging in, select View/Modify Partition in the upper left corner. Once the page refreshes, choose the LPAR you are going to modify. Select the client LPAR in the menu and be sure that your browser supports pop-up menus. On the pop-up menu, select the "Ethernet" tab (Figure 5). Figure 5. Remember to power-off LPAR to modify properties   In Figure 5, you can see that Virtual Ethernet pull-downs are grayed out; that is because the LPAR was running when the screenshot was taken. Make sure the client LPAR is powered off or inactive before modifying the properties. Use the pull-downs to map the VLAN-aware VEAs to the client LPAR's adapter on this screen. Notice that the virtual adapter is associated with one VLAN—this allows the IVM to attach VLAN tags to the traffic as it comes in from the operating systems and sends it out the appropriate LNAGG device. If more adapters are needed, click Create Adapter as needed.     Step 5: Configure Linux to use the VLAN-aware interfaces Once the IVM and Cisco switches have been configured, you may need to do one additional step if the configuration requires static IP addresses for your Linux partitions. From the IVM GUI, activate the LPAR. Log in to the box and change your user to root. Type the following: cd /etc/sysconfig/network-scripts. If you perform an ifconfig command, you will see that the virtual adapter individual VLAN mapped to the LPAR. With your favorite editor, change the interfaces parameters to meet your configuration requirements. The following is an example of an interface, ifcfg-eth0, with a static IP address: Listing 5. An interface with a static IP address   DEVICE=eth0 BOOTPROTO=static BROADCAST=192.168.1.31 HWADDR=00:1A:64:8C:B2:32 IPADDR=192.168.1.44 NETMASK=255.255.255.0 ONBOOT=yes TYPE=Ethernet GATEWAY=192.168.1.1   Restart the interfaces using /etc/init.d/network restart.     Conclusion As with any deployment, planning is crucial to success. To avoid costly rework, it's important to lay out your network design before implementing a complex network within the blade chassis. From our experience, we can tell you that reconfiguring a complex IVM-supported network requires considerable effort; in fact, the administrator usually must remove the previous configuration before reconfiguring. Planning is also critical to the install because you cannot add new VLANs to the virtual adapters on the fly in the IVM. Since you can only have one IVM in the JS22, you cannot use SEA failover as you can in a traditional VIOS installation. Link aggregation provides a mechanism for routing traffic across multiple switches in the case of a network failure of a switch. When considering redundancy in the blade, remember that the top half of the blade is powered by one PDU, while the bottom half is powered by the other PDU. All these considerations add up to a relatively complex network implementation, so our best advice is to plan before you act.