Configuration and management of slave KDC on AIX

Introduction IBM Network Authentication Service V1.4 (IBM NAS) is the IBM flavor of Kerberos, a popular network authentication protocol. AIX makes use of IBM NAS in its various components like AIX Network File System V4 (NFSv4), AIX integrated login, rlogin, telnet, openSSH, and more. Typically, IBM NAS is deployed by various enterprises that opt to use Kerberos as their network authentication protocol for their AIX systems. IBM NAS components can be categorized into three categories: IBM NAS Server component: The server component mainly consists of KDC (Key Distribution Center) and kadmind (Kerberos administration server) functionality. This server component is available only on AIX. IBM NAS Client component: This component consists of a client package, a client runtime environment, sample files, and code filesets. IBM NAS Toolkit: This component consists of the IBM Network Authentication Service application development toolkit.   All three of these components are available for IBM NAS for AIX from the AIX expansion pack CD or from the AIX Web Download Pack Programs (see Resources). For enhanced reliability, availability, and load balancing, IBM NAS KDC supports a master-slave configuration where you can have Kerberos authentication requests split across different AIX servers. Configuration of IBM NAS KDC is one of the critical tasks and so is the setup and configuration of the slave KDC. This article takes you through the configuration of the master KDC and slave KDC for IBM NAS on AIX. Example setup For this article, we used the following machine setup to install and configure IBM NAS master KDC and slave KDC. We are also using IBM NAS version 1.4.0.7, which you can get from either the AIX Expansion Pack CDs or from the IBM AIX Web Download Pack Programs (See Resources). Kerberos Administrator Name: admin/admin Kerberos Realm Name: ISL.IN.IBM.COM IBM NAS 1.4.0.7 Master KDC: Hostname: fsaix12.in.ibm.com Port: 88 OS: AIX 5.3 IBM NAS 1.4.0.7 Administration Server: Hostname: fsaix12.in.ibm.com Port: 749 OS: AIX 5.3 IBM NAS 1.4.0.7 Slave KDC: Hostname: huntcup.in.ibm.com Port: 754 OS: AIX 5.2 IBM NAS 1.4.0.7 client configured to the Master KDC: Hostname: aixdce3.in.ibm.com OS: AIX 5.3 IBM NAS 1.4.0.7 client configured to the Slave KDC (with Master KDC as a fallback mechanism): Hostname: aixdce39.in.ibm.com OS: AIX 5.3   Figure 1. Example setup for Kerberos master-slave configuration   Use the lslpp command to see which IBM NAS version is currently installed on the system. -bash-2.05b# lslpp -l | grep krb5 krb5.client.rte 1.4.0.7 COMMITTED Network Authentication Service krb5.client.samples 1.4.0.7 COMMITTED Network Authentication Service krb5.lic 1.4.0.7 COMMITTED Network Authentication Service krb5.server.rte 1.4.0.7 COMMITTED Network Authentication Service krb5.toolkit.adt 1.4.0.7 COMMITTED Network Authentication Service krb5.client.rte 1.4.0.7 COMMITTED Network Authentication Service krb5.server.rte 1.4.0.7 COMMITTED Network Authentication Service -bash-2.05b#   Also, make sure that you are using the correct IBM NAS binaries from /usr/krb5/bin and /usr/krb5/sbin. If the path is other than these two, change the PATH environment variable accordingly. -bash-2.05b# which kinit /usr/bin/kinit <---- INCORRECT IBM NAS BINARY -bash-2.05b# export PATH=/usr/krb5/bin:/usr/krb5/sbin:$PATH -bash-2.05b# which kinit /usr/krb5/bin/kinit <---- CORRECT IBM NAS BINARY -bash-2.05b#    Configuration of IBM NAS master KDC with a legacy database The Kerberos master KDC is one of the most important entities in the whole Kerberos setup. The KDC does the job of authenticating the clients (or services). It is responsible for accepting the clients’ requests for proving their identities and replying to them with the authentication information (or the error as the case may be). Since Kerberos users’ information can be stored either in the flat file on the local file system (or, "legacy database") or in the LDAP directory, the administrators need to configure the master KDC accordingly. In the following example, we cover how to set up the master KDC with the legacy database. In order to configure the Kerberos KDC, IBM NAS provides an easy-to-use command-line interface, ‘/usr/krb5/sbin/config.krb5’. This config.krb5 command can also be used to configure the slave KDC and Kerberos clients (which we will see later in this article). The syntax for the config.krb5 command is: bash-2.05b# /usr/krb5/sbin/config.krb5 To configure a server: /usr/krb5/sbin/config.krb5 -h | -S [-a admin] -d domain -r realm [[-l { ldapserver | ldapserver:port }] [-u ldap_DN -p ldap_DN_pw] [-f {keyring | keyring:entry_dn} -k keyring_pw] [-m masterkey_location] [-b bind_type] [-R ldap_replica_list]] To configure a slave KDC on a slave KDC machine: /usr/krb5/sbin/config.krb5 -h | -E -d domain -r realm -s server { [-a admin] | -l { ldapserver | ldapserver:port } -u ldap_DN -p ldap_DN_PW [-f {keyring | keyring:entry_dn} -k keyring_pw] [-b bind_type] [-R ldap_replica_list] } To configure a slave KDC on a master server machine: /usr/krb5/sbin/config.krb5 -h | -P -r realm -d domain -e slave_KDC -g To configure a client: /usr/krb5/sbin/config.krb5 -h | [-C] -r realm -d domain { -c KDC -s server | -l { ldapserver | ldapserver:port_number } [-c KDC -s server] } -bash-2.05b#   Here are the parameters that are necessary for the config.krb5 command while configuring the master KDC with a legacy database: -S denotes that we are going to configure a master KDC. -d <domain> <domain> is the name of domain where the Kerberos realm belongs. -r <realm> <realm> is the name of the Kerberos realm where this master KDC is going to reside. Make sure that before using the config.krb5 command, the hostname of the system is set to a fully qualified domain name (FQDN). To check or change the hostname on AIX, use the ‘hostname’ command. Now let's configure a master KDC with a legacy database on the fsaix12.in.ibm.com server: bash-2.05b# hostname fsaix12.in.ibm.com bash-2.05b# config.krb5 -S -d in.ibm.com -r ISL.IN.IBM.COM Initializing configuration... Creating /etc/krb5/krb5_cfg_type... Creating /etc/krb5/krb5.conf... Creating /var/krb5/krb5kdc/kdc.conf... Creating database files... Initializing database '/var/krb5/krb5kdc/principal' for realm 'ISL.IN.IBM.COM' master key name 'K/M@ISL.IN.IBM.COM' You are prompted for the database Master Password. It is important that you DO NOT FORGET this password. Enter database Master Password: <------ Enter database password to set. Re-enter database Master Password to verify: <----------- re-enter password. WARNING: no policy specified for admin/admin@ISL.IN.IBM.COM; defaulting to no policy. Note that policy may be overridden by ACL restrictions. Enter password for principal "admin/admin@ISL.IN.IBM.COM": <------ Enter Kerberos administrator password to set. Re-enter password for principal "admin/admin@ISL.IN.IBM.COM": <------ re-enter password. Principal "admin/admin@ISL.IN.IBM.COM" created. Creating keytable... Creating /var/krb5/krb5kdc/kadm5.acl... Starting krb5kdc... krb5kdc was started successfully. Starting kadmind... kadmind was started successfully. The command completed successfully. -bash-2.05b#    Here is what the above command did: Created the Kerberos configuration files (/etc/krb5/krb5.conf, /var/krb5/krb5kdc/kdc.conf and /etc/krb5/krb5_cfg_type). These configuration files are used by the IBM NAS components to get information about this Kerberos setup. Created Kerberos principal database (/var/krb5/krb5kdc/principal). This is where all the Kerberos principal and policy information is stored when configured to a legacy database. Created important Kerberos principals for us (such as admin/admin, kadmin/admin, and more). These principals are used by Kerberos administration utilities like the kadmin interface and more. Created kadmind keytab file and ACL file. These files are used by the Kerberos administration server (kadmind) for its proper execution. Started Kerberos daemons (krb5kdc and kadmind). Now the master KDC is up and running. Use the ps command to check whether the daemons are running or not. bash-2.05b# hostname fsaix12.in.ibm.com bash-2.05b# ps -ef| grep krb5 | grep -v grep root 22414 1 0 May 12 - 0:00 /usr/krb5/sbin/krb5kdc root 24210 1 0 May 12 - 0:02 /usr/krb5/sbin/kadmind bash-2.05b#    The config.krb5 command may not run due to a previous IBM NAS configuration on the system and might show something like this: bash-2.05b# config.krb5 -S -d in.ibm.com -r ISL.IN.IBM.COM Initializing configuration... /etc/krb5/krb5_cfg_type already exists. Configuration cannot be completed. A previous configuration exists. To remove the existing configuration, run the unconfig.krb5 command. bash-2.05b#    If you are sure you want to overide any existing configuration, run the ‘/usr/krb5/sbin/unconfig.krb5’ command to unconfigure any previous configuration and start afresh. bash-2.05b# unconfig.krb5 Warning: All configuration information will be removed. Do you wish to continue? [y/n] y Removing configuration... Stopping /usr/krb5/sbin/krb5kdc... /usr/krb5/sbin/krb5kdc was stopped successfully. Stopping /usr/krb5/sbin/kadmind... /usr/krb5/sbin/kadmind was stopped successfully. The command completed successfully. bash-2.05b#    To check the configuration type of the existing IBM NAS Kerberos configuration, check the contents of the /etc/krb5/krb5_cfg_type file. bash-2.05b# hostname fsaix12.in.ibm.com bash-2.05b# cat /etc/krb5/krb5_cfg_type master bash-2.05b#    For the master KDC, the file shows ‘master’. For a slave KDC, it shows ‘slave’, and for a simple client, it shows ‘client’. Note that when you configure the IBM NAS master or slave KDC, the IBM NAS client to it is already configured on that machine, so there is no need to configure the client explicitly. To check whether the master KDC is configured properly or not, try getting an initial Kerberos ticket by using /usr/krb5/bin/kinit like this: bash-2.05b# hostname fsaix12.in.ibm.com bash-2.05b# /usr/krb5/bin/kdestroy bash-2.05b# /usr/krb5/bin/kinit admin/admin Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. bash-2.05b#    To see the ticket that you have just acquired, use the /usr/krb5/bin/klist command: bash-2.05b# /usr/krb5/bin/klist Ticket cache: FILE:/var/krb5/security/creds/krb5cc_0 Default principal: admin/admin@ISL.IN.IBM.COM Valid starting Expires Service principal 05/13/08 19:40:40 05/14/08 19:40:37 krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM bash-2.05b#   Also, check that the kadmind is working properly by using the /usr/krb5/sbin/kadmin interface: bash-2.05b# /usr/krb5/sbin/kadmin -p admin/admin Authenticating as principal admin/admin with password. Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. kadmin: getprincs K/M@ISL.IN.IBM.COM admin/admin@ISL.IN.IBM.COM kadmin/admin@ISL.IN.IBM.COM kadmin/changepw@ISL.IN.IBM.COM kadmin/history@ISL.IN.IBM.COM krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM kadmin: q bash-2.05b#   Since we are able to see list of principals, the master KDC is working correctly. Now let's learn how to configure a slave KDC to this master KDC. Configuration of IBM NAS slave KDC with a legacy database In the actual production environment, there are hundreds or thousands of Kerberos clients configured to the Kerberos KDC. There may be situations when the KDC might reject the client requests due to overload of continuous requests or might even go down for some undesirable reasons. The Kerberos slave KDC comes as a rescuer in such scenarios. The slave KDC acts as a backup for the master KDC in case of emergencies like power down, network outage, fire, or other disasters. It also does the work of load balancing and reduces the burden off the master KDC. Having a slave KDC assures continuous fulfillment of client authentication requests. There can be more than one slave KDC in a Kerberos realm. It is always recommended to have at least one slave KDC for a master KDC. The distinguishing factors between the master KDC and the slave KDC is that the master KDC machine also has the administration server daemon (kadmind) running. There exists only one kadmind server in the whole realm, which resides on the master KDC machine. Also, the master KDC machine has the read-write copy of the Kerberos database (legacy database), whereas the slave KDC has a read-only copy of the Kerberos database. The configuration of the IBM NAS slave KDC requires some amount of expertise in terms of Kerberos and a better understanding of the master-slave scenario. The configuration steps consist of initial IBM NAS configuration, followed by the propagation of the Kerberos database from the master KDC to the slave KDC. Note that during the slave KDC configuration, we also need to access the master KDC machine to execute the database propagation command. To configure the slave KDC, use the same /usr/krb5/sbin/config.krb5 command, but this time with the -E option. You can use the following config.krb5 options in a slave configuration with the legacy database: -E Denotes that a slave KDC is going to be configured. -d <domain> The <domain> is the name of domain where the Kerberos realm belongs to. -r <realm> The <realm> is the name of the Kerberos realm in which the master KDC is running. -s <server> The <server> is the hostname of the machine where the master KDC and kadmind administration server are running. Now, let's go through the configuration steps. But before doing anything, please make sure that you are using a fully qualified hostname for the slave machine. On the slave machine (which is huntcup.in.ibm.com for this example) log in with the root user and run the config.krb5 command with the following options: -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# /usr/krb5/sbin/config.krb5 -E -d in.ibm.com -r ISL.IN.IBM.COM -s fsaix12.in.ibm.com Initializing configuration... Creating /etc/krb5/krb5_cfg_type... Creating /etc/krb5/krb5.conf... Creating /var/krb5/krb5kdc/kdc.conf... Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. Creating /etc/krb5/krb5.keytab... Creating /var/krb5/krb5kdc/kpropd.acl...   On your master server machine, fsaix12.in.ibm.com, run the following command: /usr/krb5/sbin/config.krb5 -P -r ISL.IN.IBM.COM -d in.ibm.com -e huntcup.in.ibm.com Have you successfully run the above command? [y/n]   At this point, the command will wait for user to input. It will ask the user to run the above ‘config.krb5 -P’ command on the master server machine. Let’s see what all the options stand for:   -P Tells the master KDC to propagate the Kerberos database to the slave KDC during the configuration. -r <realm> Specifies the name of the Kerberos <realm> where the master KDC is running. -d <domain> Speiciies the <domain> that the Kerberos realm belongs to. -e <slave_kdc_hostname> Specifies the hostname of the machine where the slave is going to reside.   Do not do anything on the slave machine for now and go to the master server machine and run the command as follows: bash-2.05b# hostname fsaix12.in.ibm.com bash-2.05b# /usr/krb5/sbin/config.krb5 -P -r ISL.IN.IBM.COM -d in.ibm.com -e huntcup.in.ibm.com Creating host/fsaix12.in.ibm.com@ISL.IN.IBM.COM... Principal "host/fsaix12.in.ibm.com@ISL.IN.IBM.COM" created. Creating /etc/krb5/krb5.keytab... Creating a dump of the database... Propagating the database to the slave KDC, huntcup.in.ibm.com ... Database propagation to huntcup.in.ibm.com: SUCCEEDED The command completed successfully. bash-2.05b#   This command tries to propagate the Kerberos database to the slave KDC. Once you get the successful message as shown, go to the slave machine and click y to continue the slave configuration process: Have you successfully run the above command? [y/n] y Unable to retrieve master key. Status 0x96c73afe - Cannot find or read the stored master key Warning: proceeding without master key. Enter database Master Password: <------ Enter database password here. Starting krb5kdc... krb5kdc was started successfully. The command completed successfully. bash-2.05b#   Once the slave configuration is completed successfully, you can check the running slave KDC by using the ps command: -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# ps -ef |grep krb5kdc | grep -v grep root 14286 1 0 15:20:17 - 0:00 /usr/krb5/sbin/krb5kdc   And also check the IBM NAS configuration type at this point: -bash-2.05b# cat /etc/krb5/krb5_cfg_type slave   To verify the working configuration, try getting the initial Kerberos ticket using the /usr/krb5/bin/kinit command: -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# /usr/krb5/bin/kinit admin/admin Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. -bash-2.05b#   Check the ticket that you have just acquired by using /usr/krb5/bin/klist command: -bash-2.05b# /usr/krb5/bin/klist Ticket cache: FILE:/var/krb5/security/creds/krb5cc_0 Default principal: admin/admin@ISL.IN.IBM.COM Valid starting Expires Service principal 05/15/08 16:29:17 05/16/08 16:29:17 krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM Renew until 05/16/08 16:29:35 -bash-2.05b#   And also check whether the administration server (which is still on the master KDC machine) is properly configured and can be contacted from slave, like this: -bash-2.05b# /usr/krb5/sbin/kadmin -p admin/admin Authenticating as principal admin/admin with password. Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. kadmin: getprincs K/M@ISL.IN.IBM.COM admin/admin@ISL.IN.IBM.COM kadmin/admin@ISL.IN.IBM.COM kadmin/changepw@ISL.IN.IBM.COM kadmin/history@ISL.IN.IBM.COM krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM kadmin: q -bash-2.05b#   This confirms that the slave KDC has been successfully configured to the master KDC. IBM NAS database propagation to slave KDC Generally, in a Kerberos environment, there are multiple slave KDCs catering the authentication requests of many clients from all over the network. Subsequently, there exist multiple copies of the Kerberos database (one per slave KDC and one for master KDC). Since the changes are only done in the master copy of the database owned by Kerberos administration server (kadmind), it's the responsibility of the master server machine to propagate the changes time to time to all the Kerberos database copies. Once all slave KDCs are in place, the Kerberos administrator needs to set up for seamless updating of the slave databases. The Kerberos database is propagated from the master KDC to the slave KDCs by using the /usr/krb5/sbin/kprop command on the master KDC and /usr/krb5/sbin/kpropd daemon on the slave KDC. This propagation requires some settings at the slave KDCs and at the master KDC. These settings are done automatically by the config.krb5 command. It is important to understand them, as it might help to troubleshoot later. Let's see these settings one by one: Host principals The propagation of the Kerberos database uses the Kerberos authentication and encryption for the secure and safe transfer. For this, each KDC needs a host principal in the Kerberos database. This host principal is created by the name ‘host/<hostname>’, where the hostname needs to be a fully qualified hostname of the slave KDC (such as huntcup.in.ibm.com). An entry of this host principal should be present in the keytab file on the slave KDC. Also, the host principal needs to be mentioned in the /var/krb5/krb5kdc/kpropd.acl file on the slave KDC. This file contains the name of the principal who is authorized to propagate the Kerberos database to that slave KDC. As we already have a slave KDC configured, we can verify that above things have already been done for us. To check the host principal entry for our slave KDC (huntcup.in.ibm.com), use the /usr/krb5/sbin/kadmin.local interface the on master KDC: bash-2.05b# hostname fsaix12.in.ibm.com bash-2.05b# kadmin.local -q getprincs K/M@ISL.IN.IBM.COM admin/admin@ISL.IN.IBM.COM host/fsaix12.in.ibm.com@ISL.IN.IBM.COM host/huntcup.in.ibm.com@ISL.IN.IBM.COM kadmin/admin@ISL.IN.IBM.COM kadmin/changepw@ISL.IN.IBM.COM kadmin/history@ISL.IN.IBM.COM krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM bash-2.05b#   And to check the keytab entry, use the /usr/krb5/bin/klist command on the slave KDC: -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# /usr/krb5/bin/klist -k Keytab name: FILE:/etc/krb5/krb5.keytab KVNO Principal ---- --------- 3 host/huntcup.in.ibm.com@ISL.IN.IBM.COM 3 host/huntcup.in.ibm.com@ISL.IN.IBM.COM 3 host/huntcup.in.ibm.com@ISL.IN.IBM.COM 3 host/huntcup.in.ibm.com@ISL.IN.IBM.COM -bash-2.05b#   While being on the slave KDC again, we can check the ACL file (like /var/krb5/krb5kdc/kpropd.acl) used by the kpropd daemon. -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# cat /var/krb5/krb5kdc/kpropd.acl host/fsaix12.in.ibm.com@ISL.IN.IBM.COM -bash-2.05b#   kpropd entry in /etc/inetd.conf and /etc/services Since the kpropd is the daemon, which is being activated whenever a request to accept propagation comes on the slave KDC, an entry of kpropd is to be made in the /etc/inetd.conf file. This way, the operating system will make sure to execute the correct binaries. The config.krb5 command automatically does this entry for us. We can verify it by grepping the /etc/inetd.conf file on the slave KDC, like this: -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# grep kprop /etc/inetd.conf krb5_prop stream tcp nowait root /usr/krb5/sbin/kpropd kpropd -bash-2.05b#   Also, a krb5_prop entry needs to be done in the /etc/services file to tell the port number on which the daemon will listen for incoming requests (754 by default). -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# grep krb5_prop /etc/services krb5_prop 754/tcp # kerberos 5 slave propagation -bash-2.05b#   So these were the prerequisites for the database propagation. Let's see the step-by-step procedure of how this database is actually propagated: First of all, the master database needs to be dumped to a file on the master KDC. For this, use the /usr/krb5/sbin/kdb5_util dump command: -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# /usr/krb5/sbin/kdb5_util dump -verbose /var/krb5/krb5kdc/krb5_dump K/M@ISL.IN.IBM.COM admin/admin@ISL.IN.IBM.COM host/fsaix12.in.ibm.com@ISL.IN.IBM.COM host/huntcup.in.ibm.com@ISL.IN.IBM.COM kadmin/admin@ISL.IN.IBM.COM kadmin/changepw@ISL.IN.IBM.COM kadmin/history@ISL.IN.IBM.COM krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM -bash-2.05b#   This creates the dump file in the specified location. We can check that out easily: -bash-2.05b# ls -lrt /var/krb5/krb5kdc/ total 104 -rw-r----- 1 root security 723 May 12 19:51 kdc.conf -rw------- 1 root system 0 May 12 19:51 principal.kadm5.lock -rw------- 1 root system 8192 May 12 19:51 principal.kadm5 -rw------- 1 root system 30 May 12 19:51 .k5.ISL.IN.IBM.COM -rw------- 1 root system 574 May 12 19:51 kadm5.keytab -rw-r----- 1 root security 135 May 12 20:25 kadm5.acl -rw------- 1 root system 16384 May 15 16:19 principal -rw------- 1 root system 0 May 15 16:19 principal.ok -rw------- 1 root system 1 May 22 16:20 krb5_dump.dump_ok -rw------- 1 root system 6960 May 22 16:20 krb5_dump -bash-2.05b#   Now we need to transfer this dump file to the slave KDC. For this, we will be using the /usr/krb5/sbin/kprop command. -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# /usr/krb5/sbin/kprop -f /var/krb5/krb5kdc/krb5_dump -d huntcup.in.ibm.com 6960 bytes sent. Database propagation to huntcup.in.ibm.com: SUCCEEDED -bash-2.05b#   At this time, the kprop command contacted the master KDC, used the host principal of master KDC (host/fsaix12.in.ibm.com) to get the service ticket for host principal of the slave KDC (host/huntcup.in.ibm.com), and once authenticated, it securely transferred the database dump file to the slave KDC. This can also be verified using the krb5kdc log file at the time of executing the kprop command; the log file (/var/krb5/krb5kdc/krb5kdc.log) should show something like this: …… …… May 22 16:28:42 fsaix12.in.ibm.com /usr/krb5/sbin/krb5kdc[22430](info): AS_REQ (5 etypes {16 23 18 3 1}) 9.182.192.35(88): ISSUE: authtime 1211452122, etypes {rep=16 tkt=16 ses=16}, host/fsaix12.in.ibm.com@ISL.IN.IBM.COM for host/huntcup.in.ibm.com@ISL.IN.IBM.COM   That’s all it takes to propagate your Kerberos principal database to your slave KDC. Though very simple, this could be a hectic task if there are more slave KDCs needing the propagation. A great idea would be to write a simple script that does the propagation job to all the slave KDCs for us and even greater would be to add this script to a cron job on the master KDC, so that this can happen periodically. In the next section, we will see how to achieve this. Simple Kerberos database propagation script Here is a very simple shell script that you can use right away for the database propagation. #!/bin/sh KDCLIST="slave1.in.ibm.com slave2.in.ibm.com slave3.in.ibm.com" DUMP_FILE="/var/krb5/krb5kdc/krb5_dump" /usr/krb5/sbin/kdb5_util dump $DUMP_FILE for kdc in $KDCLIST do /usr/krb5/bin/kprop -f $DUMP_FILE $kdc done -bash-2.05b#   This script creates the database dump and transfers it to all the mentioned slave KDCs. Please replace the slave1.in.ibm.com, slave2.in.ibm.com and etc. with the actual hostnames of your slave KDCs. To run this shell script as a cron job, use the following the AIX command crontab, as shown: -bash-2.05b# crontab -e   This gives you a text editor ('vi' by default) to add your cron job. You will see all the already-present cron jobs, so add your cron job at the end. Enter the following text to add your job: 59 23 * * * /var/krb5/krb5kdc/prop_krb5_database.sh   Save your entry by clicking :wq! (a vi command to save and exit) and you will be back to the prompt. Adding this line runs the script ‘/var/krb5/krb5kdc/prop_krb5_database.sh’ at 11:59 pm every day. For more information on cron job commands and options, see the main pages of the crontab AIX command. In order to know how to determine the frequency of the Kerberos database propagation (for legacy databases only), please refer the IBM NAS Version 1.4 Administration Guide, which ships with the AIX Version 5.3 Expansion Pack CD. To verify your entry in the cron table, use the crontab -l command. On our master KDC machine, it shows something like this: -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# crontab -l 0 11 * * * /usr/bin/errclear -d S,O 30 0 12 * * * /usr/bin/errclear -d H 90 0 15 * * * /usr/lib/ras/dumpcheck >/dev/null 2>&1 # SSA warning : Deleting the next two lines may cause errors in redundant # SSA warning : hardware to go undetected. 01 5 * * * /usr/lpp/diagnostics/bin/run_ssa_ela 1>/dev/null 2>/dev/null 0 * * * * /usr/lpp/diagnostics/bin/run_ssa_healthcheck 1>/dev/null 2>/dev/null # SSA warning : Deleting the next line may allow enclosure hardware errors to go undetected 30 * * * * /usr/lpp/diagnostics/bin/run_ssa_encl_healthcheck 1>/dev/null 2>/dev/null # SSA warning : Deleting the next line may allow link speed exceptions to go undetected 30 4 * * * /usr/lpp/diagnostics/bin/run_ssa_link_speed 1>/dev/null 2>/dev/null 0 00,12 * * * wall%rc.powerfail:1::WARNING!!! The system is now operating with cooling problem. This message will be walled every 12 hours. Remove this crontab entry after the problem is resolved. 59 23 * * * /var/krb5/krb5kdc/prop_krb5_database.sh -bash-2.05b#   That's all it takes to automate the database propagation task. Now that we have the master KDC and slave KDC ready, next we will learn how to configure clients to these machines. Setting up IBM NAS Kerberos clients Once the Kerberos master KDC and slave KDCs are in place, the next task to do is to configure and set up Kerberos clients to these KDCs. Depending upon your requirement, you can configure the clients either directly to the master KDC or to the slave KDC with master KDC as a fallback. We will see both the methods. First let’s configure a client to our master KDC (running at fsaix12.in.ibm.com). Configuring an IBM NAS client directly to the master KDC To configure an IBM NAS client, we will again use the config.krb5 command but with a ‘-C’ flag and some other flags. Let's have a look at the flags used in configuring a client: -C Indicates that an IBM NAS client is going to be configured. -d <domain> Specifies the name of the Kerberos realm <domain>. -r <realm> Specifies the Kerberos <realm> iwhere the master KDC is running. -c <KDC> Specifies the hostname of the machine where the master KDC server is running. -s <server> Specifies the hostname of the machine where the kadmind administration server is running. Before running this command, make sure that the hostname of the client machine is fully qualified and the current date and time is in sync with the master KDC. We are using aixdce3.in.ibm.com as our client machine. Now, log in as root and run the config.krb5 command with the appropriate options: bash-2.05b# hostname aixdce3.in.ibm.com bash-2.05b# config.krb5 -C -d in.ibm.com -r ISL.IN.IBM.COM -c fsaix12.in.ibm.com -s fsaix12.in.ibm.com Initializing configuration... Creating /etc/krb5/krb5_cfg_type... Creating /etc/krb5/krb5.conf... The command completed successfully. bash-2.05b#   The above client setup creates the IBM NAS configuration files and no more. As expected, this time the /etc/krb5/krb5_cfg_type file will contain ‘client’ to denote the client configuration. The IBM NAS client configuration file /etc/krb5/krb5.conf file contains the Kerberos information, such as: bash-2.05b# hostname aixdce3.in.ibm.com bash-2.05b# cat /etc/krb5/krb5.conf [libdefaults] default_realm = ISL.IN.IBM.COM default_keytab_name = FILE:/etc/krb5/krb5.keytab default_tkt_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc default_tgs_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc [realms] ISL.IN.IBM.COM = { kdc = fsaix12.in.ibm.com:88 admin_server = fsaix12.in.ibm.com:749 default_domain = in.ibm.com } [domain_realm] .in.ibm.com = ISL.IN.IBM.COM fsaix12.in.ibm.com = ISL.IN.IBM.COM [logging] kdc = FILE:/var/krb5/log/krb5kdc.log admin_server = FILE:/var/krb5/log/kadmin.log default = FILE:/var/krb5/log/krb5lib.log bash-2.05b#   Please note that both ‘kdc’ and ‘admin_server’ entries point to the same master KDC machine here (because both the Kerberos daemons are running there). To verify the client setup, you need to use the ‘/usr/krb5/bin/kinit’ and ‘/usr/krb5/bin/klist’ commands, as shown below: bash-2.05b# hostname aixdce3.in.ibm.com bash-2.05b# /usr/krb5/bin/kinit admin/admin Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. bash-2.05b# /usr/krb5/bin/klist Ticket cache: FILE:/var/krb5/security/creds/krb5cc_0 Default principal: admin/admin@ISL.IN.IBM.COM Valid starting Expires Service principal 05/22/08 09:46:13 05/23/08 09:43:53 krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM bash-2.05b#   Also, check that the client can contact the kadmind server by running the kadmind interface: bash-2.05b# /usr/krb5/sbin/kadmin -p admin/admin Authenticating as principal admin/admin with password. Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. kadmin: getprincs K/M@ISL.IN.IBM.COM admin/admin@ISL.IN.IBM.COM host/fsaix12.in.ibm.com@ISL.IN.IBM.COM host/huntcup.in.ibm.com@ISL.IN.IBM.COM kadmin/admin@ISL.IN.IBM.COM kadmin/changepw@ISL.IN.IBM.COM kadmin/history@ISL.IN.IBM.COM krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM kadmin: q bash-2.05b#   Now, our Kerberos client has been configured, the client was directly configured to the master KDC. In the next section, we will see how to configure a client to the slave KDC with master KDC as a fallback mechanism. Configuring an IBM NAS client to the slave KDC (with master KDC as a fallback mechanism) The configuration steps in this case are going to be similar to the previous one. This time, the client machine will be ‘aixdce39.in.ibm.com’. While running the config.krb5 command, all the options are same as the previous example. The only difference would be that this time with -c, we will specify the hostname of the slave KDC machine, not the master KDC. Let's configure the client on this machine: [root@aixdce39 / ] #hostname aixdce39.in.ibm.com [root@aixdce39 / ] #config.krb5 -C -d in.ibm.com -r ISL.IN.IBM.COM -c huntcup.in.ibm.com -s fsaix12.in.ibm.com Initializing configuration... Creating /etc/krb5/krb5_cfg_type... Creating /etc/krb5/krb5.conf... The command completed successfully. [root@aixdce39 / ] #   We will check the /etc/krb5/krb5.conf file: [root@aixdce39 / ] #cat /etc/krb5/krb5.conf [libdefaults] default_realm = ISL.IN.IBM.COM default_keytab_name = FILE:/etc/krb5/krb5.keytab default_tkt_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc default_tgs_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc [realms] ISL.IN.IBM.COM = { kdc = huntcup.in.ibm.com:88 admin_server = fsaix12.in.ibm.com:749 default_domain = in.ibm.com } [domain_realm] .in.ibm.com = ISL.IN.IBM.COM fsaix12.in.ibm.com = ISL.IN.IBM.COM huntcup.in.ibm.com = ISL.IN.IBM.COM [logging] kdc = FILE:/var/krb5/log/krb5kdc.log admin_server = FILE:/var/krb5/log/kadmin.log default = FILE:/var/krb5/log/krb5lib.log [root@aixdce39 / ] #   This time, our client is talking to the slave KDC (huntcup.in.ibm.com). Verify the configuration with the /usr/krb5/bin/kinit and /usr/krb5/bin/klist commands: [root@aixdce39 / ] #hostname aixdce39.in.ibm.com [root@aixdce39 / ] #/usr/krb5/bin/kinit admin/admin Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. [root@aixdce39 / ] #/usr/krb5/bin/klist Ticket cache: FILE:/var/krb5/security/creds/krb5cc_0 Default principal: admin/admin@ISL.IN.IBM.COM Valid starting Expires Service principal 05/22/08 20:12:38 05/23/08 20:12:25 krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM [root@aixdce39 / ] #   Also, we can check the kadmin interface using the /usr/krb5/sbin/kadmin command: [root@aixdce39 / ] #/usr/krb5/sbin/kadmin -p admin/admin Authenticating as principal admin/admin@ISL.IN.IBM.COM with password. Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. kadmin: getprincs K/M@ISL.IN.IBM.COM admin/admin@ISL.IN.IBM.COM host/fsaix12.in.ibm.com@ISL.IN.IBM.COM host/huntcup.in.ibm.com@ISL.IN.IBM.COM kadmin/admin@ISL.IN.IBM.COM kadmin/changepw@ISL.IN.IBM.COM kadmin/history@ISL.IN.IBM.COM krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM kadmin: q [root@aixdce39 / ] #   Now that everything is working fine, we want our client to fallback to the master KDC if something goes wrong with the slave KDC. In order to do that, we need to modify the /etc/krb5/krb5.conf file. Edit this file and add kdc = <master_KDC_hostname> as shown below: [root@aixdce39 / ] #cat /etc/krb5/krb5.conf [libdefaults] default_realm = ISL.IN.IBM.COM default_keytab_name = FILE:/etc/krb5/krb5.keytab default_tkt_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc default_tgs_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc [realms] ISL.IN.IBM.COM = { kdc = huntcup.in.ibm.com:88 kdc = fsaix12.in.ibm.com:88 admin_server = fsaix12.in.ibm.com:749 default_domain = in.ibm.com } [domain_realm] .in.ibm.com = ISL.IN.IBM.COM fsaix12.in.ibm.com = ISL.IN.IBM.COM huntcup.in.ibm.com = ISL.IN.IBM.COM [logging] kdc = FILE:/var/krb5/log/krb5kdc.log admin_server = FILE:/var/krb5/log/kadmin.log default = FILE:/var/krb5/log/krb5lib.log [root@aixdce39 / ] #    The purpose of adding two ‘kdc’ entries in the /etc/krb5/krb5.conf is that this client tries to contact the first listed KDC, then the second-listed KDC if the first one is not available. You can specify more KDCs if you want, but remember that the first working entry is used. In order to check the fallback mechanism, shut down the slave KDC on huntcup.in.ibm.com, so that this client is forced to use the master KDC on fsaix12.in.ibm.com. To start and stop the IBM NAS daemons, you can use the /usr/krb5/sbin/start.krb5 and /usr/krb5/sbin/stop.krb5 commands. To find out more on these, refer to the IBM NAS Version 1.4 Administration Guide, which ships with the AIX Version 5.3 Expansion Pack CD. Run the following command to stop the slave KDC: -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# stop.krb5 krb5kdc Stopping /usr/krb5/sbin/krb5kdc... /usr/krb5/sbin/krb5kdc was stopped successfully. The command completed successfully. -bash-2.05b#   Try running the /usr/krb5/bin/kinit command from the client. [root@aixdce39 / ] #hostname aixdce39.in.ibm.com [root@aixdce39 / ] #/usr/krb5/bin/kinit admin/admin Password for admin/admin@ISL.IN.IBM.COM: <------ Enter Kerberos administrator password here. [root@aixdce39 / ] #/usr/krb5/bin/klist Ticket cache: FILE:/var/krb5/security/creds/krb5cc_0 Default principal: admin/admin@ISL.IN.IBM.COM Valid starting Expires Service principal 05/22/08 20:38:03 05/23/08 20:38:03 krbtgt/ISL.IN.IBM.COM@ISL.IN.IBM.COM Renew until 05/23/08 20:38:09 [root@aixdce39 / ] #   So, even if the slave KDC is down, the client can connect to the master KDC and continue running seamlessly. Conclusion This article, part one of a three-part series, has shown the configuration and use of the IBM NAS slave KDC with the legacy database setup. Part 2 will cover how to update the slave KDC to the master KDC. Part 3 will educate how to configure the master-slave KDC with LDAP as the back end for storing Kerberos data.