Speaking UNIX: More shell scripting techniques

If you've worked on IBM® AIX®, another flavor of UNIX®, or Linux®, you've more than likely used the vi editor. Since its conception in 1976, vi has become a staple for anyone wanting to edit files. How could someone make a more powerful editing tool than vi, you may ask? The answer is Vim, and this article provides details on the many enhancements that have made Vim a highly used and acceptable editor in the world of UNIX and Linux.       Like other UNIX operating systems and Linux, the IBM AIX operating system has several powerful tools that arm systems administrators, developers, and users to tackle day-to-day tasks and to simplify their or their customers' business and life. One such tool in UNIX is the ability to write shell scripts to automate tasks, simplifying difficult or long and tedious jobs. Although some who have worked on UNIX for a couple of years have dabbled in shell scripting, they are still most likely learning the ins and outs of the operating system and haven't yet mastered scripting. This article provides tips to those wanting to learn more about shell scripting and how to begin writing more advanced scripts. It provides basic fundamentals of programming in general, including how to simplify the script, how to keep the script as flexible as possible, how to write a clean script, documenting inside the script, and debugging the script. Keep it simple One issue people run into when learning how to shell script well is duplicating work they've already done in a script. Rather than copying their work and changing a couple of hard-coded values, they could simply create a function to handle the work for both areas of the script. Creating centralized functions also standardizes and provides for a uniform script. If a function works in one area of the script, it's a safe bet it will work elsewhere in the script, as well. For example, the script shown in Listing 1 should be condensed and simplified considerably into a smaller and more clean-looking program. Listing 1. Example of a script that can be simplified   #!/usr/bin/ksh if [[ $# -lt 2 ]] then echo "Usage: ${0##*/} <file name #1> <file name #2> exit 0 fi if [[ ! -f "${1}" ]] then echo "Unable to find file '${1}'" exit 1 fi if [[ ! -r "${1}" ]] then echo "Unable to read file '${1}'" exit 2 fi gzip ${1} ls -l ${1}.gz if [[ ! -f "${2}" ]] then echo "Unable to find file '${2}'" exit 1 fi if [[ ! -r "${2}" ]] then echo "Unable to read file '${2}'" exit 2 fi gzip ${2} ls -l ${2}.gz   This script looks awful! (Thankfully, it's only an example.) The script should be condensed as much as possible. To ease the eyes of readers, Listing 2 provides a cleaner version. Listing 2. Condensed example of the Listing 1 script   #!/usr/bin/ksh exit_msg() { [[ $# -gt 1 ]] && echo "${0##*/} (${1}) - ${2}" exit ${1:-0} } [[ $# -lt 2 ]] && exit_msg 0 "Usage: ${0##*/} <file name #1> <file name #2> for _FNAME in $@ do [[ ! -f "${_FNAME}" ]] && exit_msg 1 "Unable to find file '${_FNAME}'" [[ ! -r "${_FNAME}" ]] && exit_msg 2 "Unable to read file '${_FNAME}'" gzip ${_FNAME} ls -l ${_FNAME}.gz done   Notice the differences? By adding a simple function to display a message and exit with the appropriate return code as well as moving everything into a for loop, the script looks cleaner and is easier to understand.     Keep it flexible Another issue that novices to programming and shell scripting encounter is hard-coding static values into a program or shell script. This limits the flexibility of a script and, in short, is bad programming. To keep administrators or developers from constantly having to modify a script to work a certain way with other values, use variables and supply arguments to the script or function. For example, Listing 3 is an example of a poorly written and inflexible script. Listing 3. Example of a script that isn't flexible   #!/bin/bash if [[ -f /home/cormany/FileA ]] then echo "Found file '/home/cormany/FileA'" elif [[ -f /home/cormany/DirA/FileA ]] then echo "Found file '/home/cormany/DirA/FileA'" else echo "Unable to find file FileA" fi   This script works but is limited to searching for a single file in only two locations. To expand on the idea, the script in Listing 4 provides the same feel but allows users to search for any file in any location. Listing 4. Making a script more flexible   #!/bin/bash exit_msg() { [[ $# -gt 1 ]] && echo "${0##*/} (${1}) - ${2}" exit ${1:-0} } [[ $# -lt 2 ]] && exit_msg 1 "Usage: ${0##*/} <file name> <location>" _FNAME="${1}" _DNAME="${2}" [[ ! -d "${_DNAME}" ]] && exit_msg 2 "Unable to read or find directory '${_DNAME}'" if [[ -f "${_DNAME}/${_FNAME}" ]] then exit_msg 0 "Found file '${_DNAME}/${_FNAME}'" else exit_msg 3 "Unable to find file '${_DNAME}/${_FNAME}'" fi   This example is more flexible, as it allows users to enter any file they want to search for in any directory.     Give them options When writing a shell script, some users may say, "It would be great to have this!" or "I'd love to be able to do that!" while others may not necessarily agree and may not want to perform the same action. People love options, so why not give them some? The built-in shell command getopt does just that job. Listing 5 provides a basic example of how getopt works in AIX. Listing 5. getopt example   #!/usr/bin/ksh _ARGS=`getopt -o x --long xxxxx -n ${0##*/} -- "$@"` while [[ $# -gt 0 ]] do case "${1}" in -x|--xxxxx) echo "Arg x hit!"; shift;; --) shift; break;; *) echo "Invalid Option: ${1}"; break;; esac done   When executing the script containing getopt, named opttest, using a valid argument of -x-or --xxxxx, getopt recognizes the switch and executes the code within the case switch: # ./hm -x Arg x hit!   And again with an invalid switch or option: # ./hm -a Invalid Option: -a     Document, document, document We all fall prey to this issue at one point in our careers. You've been asked to look at a script that was written 10 years ago by someone who no longer works for the company. No problem, you say? Typically, it won't be an issue, but if the script is complex, executes commands you’re not accustom to, is written in a different style than you're use to, or just doesn't work, it's extremely helpful to get some hints as to what the person was thinking when the script was originally created. Other times, you may be the person who developed a script that you thought was to be used once and never again. Or maybe you wrote a huge script that you've worked on for weeks and know inside and out, but if someone else looked at it, that person would be completely confused. These are just a few examples of why documentation is sometimes as important to the developer as the script is to the user. Take the function shown in Listing 6 from a snippet of code. Listing 6. Example of a script with no comments   confirm_and_exit() { [[ ${_DEBUG_LEVEL} -ge 3 ]] && set -x while [[ -z ${_EXIT_ANS} ]] do cup_echo "Are you sure you want to exit? [Y/N] \c" ${_PROMPT_ERR_ROW} ${_PROMPT_ERR_COL} ${_TPUT_CMD} cnorm read ${_NO_EOL_FLAG:+${_READ_FLAG:-'-n'}} ${_NO_EOL_FLAG} _EXIT_ANS ${_TPUT_CMD} civis done case ${_EXIT_ANS} in [Nn]) unset _EXIT_ANS; return 0;; [Yy]) exit_msg 0 1 "Exiting Script";; *) invalid_selection ${_EXIT_ANS}; unset _EXIT_ANS;; esac return 0 }   If you've worked with shell scripting for a while, you may be able to read this. However, someone just learning scripting will look at this and not know what this function is doing. Taking a few extra minutes to add comments in the script can make a world of difference. Listing 7 shows the same function with comments. Listing 7. Example of a script with comments   ######################################### # function confirm_and_exit ######################################### confirm_and_exit() { # if the debug level is set to 3 or higher, send every evaluated line to stdout [[ ${_DEBUG_LEVEL} -ge 3 ]] && set –x # Continue to prompt the user until they provide a valid answer while [[ -z ${_EXIT_ANS} ]] do # prompt user if they want to exit the script # cup_echo function calls tput cup <x> <y> # syntax: # cup_echo <string to display> <row on stdout to display> <column on stdout to display> cup_echo "Are you sure you want to exit? [Y/N] \c" ${_PROMPT_ERR_ROW} ${_PROMPT_ERR_COL} # change cursor to normal via tput ${_TPUT_CMD} cnorm # read value entered by user # if _NO_EOL_FLAG is supplied, use value of _READ_FLAG or “-n” # if _NO_EOL_FLAG is supplied, use value as characters aloud on read # assign value entered by user to variable _EXIT_ANS read ${_NO_EOL_FLAG:+${_READ_FLAG:-'-n'}} ${_NO_EOL_FLAG} _EXIT_ANS # change cursor to invisible via tput ${_TPUT_CMD} civis done # if user entered “n”, return to previous block of code with return code 0 # if user entered “y”, exit the script # if user entered anything else, execute function invalid_selection case ${_EXIT_ANS} in [Nn]) unset _EXIT_ANS; return 0;; [Yy]) exit_msg 0 1 "Exiting Script";; *) invalid_selection ${_EXIT_ANS}; unset _EXIT_ANS;; esac # exit function with return code 0 return 0 }   This may seem tedious and a bit of overkill for such a small function, but the value of the comments to a novice shell scripter or someone just looking at the function can be invaluable. Another extremely helpful use for comments in shell scripts is to explain what variables may be as well as explaining return codes. The example in Listing 8 is from the beginning of a shell script. Listing 8. Example of variables not documented   #!/usr/bin/bash trap 'exit_msg 1 0 "Signal Caught. Exiting..."' HUP INT QUIT KILL ABRT trap 'window_size_changed' WINCH _MSG_SLEEP_TIME=3 _RETNUM_SIZE=6 _DEBUG_LEVEL=0 _TMPDIR="/tmp" _SP_LOG="${0##*/}.log" _SP_REQUESTS="${HOME}/sp_requests" _MENU_ITEMS=15 LESS="-P LINE\: %l" export _SP_REQUESTS _TMPDIR _SP_LOG _DB_BACKUP_DIR export _DEBUG_LEVEL _NEW_RMSYNC _RMTOTS_OFFSET_COL   Again, it's quite difficult to understand what the traps are suppose to do or what the values are for in each variable. Unless you follow through a script completely, these variables mean nothing. In addition, there's no mention of any return codes used in the script. This can make troubleshooting a problem in a shell script much more difficult than it really needs to be. Adding a few comments to the lines in Listing 8 and adding a section on which return codes are used and their descriptions cuts down on confusion immensely. Take a look at Listing 9 below. Listing 9. Example of variables documented   #!/usr/bin/bash ######################################################################### # traps ######################################################################### # trap when a user is attempting to leave the script trap 'exit_msg 1 0 "Signal Caught. Exiting..."' HUP INT QUIT KILL ABRT trap 'window_size_changed' WINCH # trap when a user has resized the window ######################################################################### ######################################################################### # defined/exported variables ######################################################################### _MSG_SLEEP_TIME=3 # seconds to sleep for all messages # (if not defined, default will is 1 second) _CUSTNUM_SIZE=6 # length of a customer number in this location # (if not defined, default is 6) _DEBUG_LEVEL=0 # log debug messages. log level is accumulative # (i.e. 1 = 1, 2 = 1 & 2, 3 = 1, 2, & 3) # (if not defined, default is 0) # Log levels: # 0 = No messages # 1 = brief messages (start script, errors, etc) # 2 = environment setup (set / env) # 3 = set -x (A LOT of spam) _TMPDIR="/tmp" # directory to put work/tmp files # (if not defined, default is /tmp) _SP_LOG="${0##*/}.log" # log of script events _SP_REQUESTS="${HOME}/sp_requests" # file to customer record requests, # also read at startup _MENU_ITEMS=15 # default number of items to display per page # (it not defined, default is 10) LESS="-P LINE\: %l" # format 'less' prompt. MAN less if more info # export the variables defined above export _MSG_SLEEP_TIME _CUSTNUM_SIZE _DEBUG_LEVEL _TMPDIR _SP_LOG _SP_REQUESTS _MENU_ITEMS #########################################################################   Doesn't that look better? Everything is organized and detailed, and someone reading the script for the first time will have a better chance in understanding what the program is doing.     Debugging You've finished writing the script, and now it's time to run the program for the first time. But when you execute the script, some unexpected errors are displayed. Now what? No one is perfect, and writing scripts from scratch to the point of not getting errors takes a lot of time and experience—most of the time, even that won't save someone from easily missing a character or transposing a couple of characters. Don't worry: The shells in AIX and other flavors of UNIX and Linux have it covered and can help with debugging. For example, the shell script in Listing 10, named make_errors (and rightfully so), has been written and is ready to be executed. Listing 10. Example of a script with errors   #!/bin/bash _X=1 while [[ ${_X} -le 10 ]] do [[ ${_X} -lt 5 ]] && echo "X is less than 5! _Y=`expr ${_X) + 1` if [[ ${_Y} -eq 6 ]] echo "Y is now equal to ${_Y}" fi _X=${_Y} done   However, when executing the script initially, the following error is displayed: # ./make_errors ./make_errors: line 11: unexpected EOF while looking for matching `"' ./make_errors: line 16: syntax error: unexpected end of file   One awesome debugging tool that you're already using and didn't know is Vim. Vim is a powerful text editor but also lends a helpful hand in debugging. If your .exrc or .vimrc file is set up to display colors for certain error conditions, Vim will do most of the work for you, as Figure 1, below, shows. Figure 1. Debugging with Vim   The first error (line 11: unexpected EOF while looking for matching `"') says something is going on with line 11, but after looking at the line, nothing looks wrong with it. Take a look at line 9. A double quotation mark (") is missing at the end of the string being echo-ed. This is a good example why you must look at the script as a whole when debugging. The line number displayed may not always be where the actual error originated. Line 11 is reporting an error because line 9 started to encapsulate a string with double quotation marks, but the string was not fully encapsulated until line 11. To correct the error, add double quotation marks to the end of line 9. Something else is blatantly showing up as an error, as well. In line 11, after the value of variable _X is a closed parenthesis ()) highlighted in red. This is Vim doing your job for you and telling you that there's something wrong. It looks like the value of variable _X started with an opening curly bracket ({) but didn't end correctly with the closing curly bracket (}). Simply changing ) to } should do the trick. Nice job: Two errors fixed so far. Run the script again and see what happens: ./make_errors: line 12: syntax error near unexpected token `fi' ./make_errors: line 12: ` fi'   Yet another error. This error says that there's a problem in line 12, but the line only has a fi completing the if statement. What's wrong with that? Keep in mind what happened with the previous error. Not all errors originate at the line at which the shell reports the problem. The shell is simply reporting where an error occurred, but this could mean that the cause of the error started before the shell reported a failure. It's a safe bet that in a script this small, the error may be in the actual if statement. Thinking back to basic shell scripting logic, an if statement consists of if, then, and fi. Looking at the conditional statement, it looks like there's a missing then. Simply add the then into the script. When you're finished, the script should look like Listing 11. Listing 11. Corrected script from Listing 10   #!/bin/bash _X=1 while [[ ${_X} -le 10 ]] do [[ ${_X} -lt 5 ]] && echo "X is less than 5!" _Y=`expr ${_X} + 1` if [[ ${_Y} -eq 6 ]] then echo "Y is now equal to ${_Y}" fi _X=${_Y} done   Run the script one more time: # ./make_errors X is less than 5! X is less than 5! X is less than 5! X is less than 5! Y is now equal to 6   Congratulations! The script now works as expected. The set -x option Sometimes, performing the basic troubleshooting steps of a shell script isn't as easy as it was in the above example. If all else fails, you're banging your head against the wall, and have no idea why the script is failing, the final step is to pull out the big guns! Ksh, Bash, and other modern shells include the switch -x in their set commands. Using the set –x option, every command evaluated is expanded and displayed to stdout. To make the evaluated code stand out, set –x uses the value of the PS4 variable and prepends it to every line of code displayed. Keep in mind this can be a lot of text to absorb, so be patient when looking through it. Tone down the loop count on the previous example, add set -x to the beginning of the script as well as a comment, and execute it, as shown in Listing 12. Listing 12. Example of set -x   #!/bin/bash set -x # loop through and display some test statements _X=1 while [[ ${_X} -le 4 ]] do [[ ${_X} -lt 2 ]] && echo "X is less than 2! _Y=`expr ${_X} + 1` if [[ ${_Y} -eq 3 ]] then echo "Y is now equal to ${_Y}" fi _X=${_Y} done   Before you execute the script, change PS4 to something that will stand out: # export PS4="DEBUG => "   Next, spam yourself with possibly very valuable information, as shown in Listing 13. Listing 13. Example output from set -x   # ./make_errors DEBUG => _X=1 DEBUG => [[ 1 -le 4 ]] DEBUG => [[ 1 -lt 2 ]] DEBUG => echo 'X is less than 2!' X is less than 2! DDEBUG => expr 1 + 1 DEBUG => _Y=2 DEBUG => [[ 2 -eq 3 ]] DEBUG => _X=2 DEBUG => [[ 2 -le 4 ]] DEBUG => [[ 2 -lt 2 ]] DDEBUG => expr 2 + 1 DEBUG => _Y=3 DEBUG => [[ 3 -eq 3 ]] DEBUG => echo 'Y is now equal to 3' Y is now equal to 3 DEBUG => _X=3 DEBUG => [[ 3 -le 4 ]] DEBUG => [[ 3 -lt 2 ]] DDEBUG => expr 3 + 1 DEBUG => _Y=4 DEBUG => [[ 4 -eq 3 ]] DEBUG => _X=4 DEBUG => [[ 4 -le 4 ]] DEBUG => [[ 4 -lt 2 ]] DDEBUG => expr 4 + 1 DEBUG => _Y=5 DEBUG => [[ 5 -eq 3 ]] DEBUG => _X=5 DEBUG => [[ 5 -le 4 ]]   As you can see, there's a lot of information here: Every single command has been evaluated and executed. Also notice that the comment place in the shell script was not displayed in the debug information. This is because the string was a comment and therefore not executed after evaluation. Thankfully, nothing was wrong with this script after you made the original corrections! One thing to keep in mind when using set -x is that if the script you're evaluating has internal functions, set -x will carry over to its child function if placed in the root body of the code. However, if set -x is placed only in the internal function, only code and child functions called within the internal function will carry in the debug option; the root body of the shell script will not, as it knows nothing of its child function calling the routine.     Conclusion We're always improving on our method of programming, regardless of whether it's shell script, C, the Java™ language, or another language someone is working on. Stick to the basic rules of simplifying the work, keeping it clean and flexible, and documenting your code, and soon you'll be writing top-notch shell scripts with a little help from the debugging method you've learned. Good luck!