Learn 10 more good UNIX usage habits

As a follow-up to Michael Stutz's excellent article, this article provides 10 more good habits to adopt that will improve your UNIX® command-line efficiency. Learn about common errors and how to overcome them, and discover exactly why these 10 UNIX habits are worth picking up!       Let's face it: Bad habits are hard to break. But habits that you've just become comfortable with can be even more difficult to overcome. Sometimes, a fresh look at things may provide you with an "A-ha, I didn't know you could do that!" moment. Building on Michael Stutz's excellent article, "Learn 10 good UNIX usage habits," this article suggests 10 more UNIX command-line commands, tools, and techniques that may make you more productive as a UNIX command-line wizard. The 10 additional good habits you should adopt are: Use file name completion. Use history expansion. Reuse previous arguments. Manage directory navigation with pushd and popd. Find large files. Create temporary files without an editor. Use the curl command-line utility. Make the most of regular expressions. Determine the current user. Process data with awk. Use file name completion Wouldn't it be great if you didn't have to type a long, convoluted file name at the command prompt? Well, you don't, as it turns out. You can configure the most popular UNIX shells for file name completion, instead. This functionality works a bit differently in each shell, so I show you how to use file name completion in the most popular shells. File name completion allows you to type faster and avoid errors. Lazy? Perhaps. More efficient? Definitely! Which shell am I running? What happens if you don’t know which shell you're currently running? Although this trick isn't officially part of the 10 more good habits, it's still pretty useful. As shown in Listing 1, you can use the echo $0 or ps –p $$ command to display the shell you're using. In my case, I'm running the Bash shell. Listing 1. Determine your shell   $ echo $0 -bash $ ps –p $$ PID TTY TIME CMD 6344 ttys000 0:00.02 –bash   C shell The C shell supports the most straightforward file name completion. Setting the filec variable enables the functionality. (You can use the command set filec). After you start typing the name of a file, you can click Escape, and the shell fills in the name of the file—or as much as it can. For example, say you have files named file1, file2, and file3. If you type f, then click Escape, file will be filled out, and you'll have to type the 1, 2, or 3 to complete the appropriate file name. Bash The Bash shell also provides file name completion but uses the Tab key instead of the Escape key. You don't need to set anything to enable file name completion in the Bash shell; it's set by default. Bash also implements an additional feature. After typing a portion of a file name, then clicking Tab, if you reach that point at which multiple files satisfy your request and you need to add text to select one of the files, you can click Tab twice more for a list of the files that match what you have typed so far. Using the earlier examples of files named file1, file2, and file3, start by typing f. When you click Tab once, Bash completes file; clicking Tab one more time expands the list of file1 file2 file3. Korn shell For Korn shell users, file name completion depends on the value of the EDITOR variable. If EDITOR is set to vi, you type part of name, and then click Escape followed by a backslash (\) character. If EDITOR is set to emacs, you type part of the name, and then click the Escape key twice to complete the file name. Use history expansion What happens if you're using the same file name for a series of commands? Well, there's a shortcut that can quickly retrieve the last file name you used. As shown in Listing 2, the !$ command returns the file name that the previous command used. The file this-is-a-long-lunch-menu-file.txt is searched for occurrences of the word pickles. After searching, the vi command is used to edit the this-is-a-long-lunch-menu-file.txt file without the need for retyping the file name. You use the bang, or exclamation point (!), to access the history, and the dollar sign ($) returns the last field of the previous command. It's a great tool if you are using long file names repeatedly. Listing 2. Using !$ to retrieve the last file name used with a command   $ grep pickles this-is-a-long-lunch-menu-file.txt pastrami on rye with pickles and onions $ vi !$   Reuse previous arguments The !$ command returns the last argument used with a command. But what happens if you have a command that used arguments and you want to reuse just one of them? The !:1 operator returns the argument used in a command. The example in Listing 3 shows how you can use this operator in combination with the !$ operator. In the first command, a file is renamed to a more meaningful name, but to preserve use of the original file name, a symbolic link is created. The file kxp12.c is renamed in a more readable manner, then the link command is used to create a symbolic link back to the original file name, in case it's still used elsewhere. The !$ operator returns the file_system_access.c argument, and the !:1 operator returns the kxp12.c argument, which is the first argument of the previous command. Listing 3. Using !$ and !:1 in combination   $ mv kxp12.c file_system_access.c $ ln –s !$ !:1   Manage directory navigation with pushd and popd UNIX supports a wide variety of directory-navigation tools. Two of my favorite productivity tools are pushd and popd. You're certainly aware that the cd command changes your current directory. What happens if you have several directories to navigate, but you want to be able to quickly return to a location? The pushd and popd commands create a virtual directory stack, with the pushd command changing your current directory and storing it on the stack, and the popd command removing the directory from the top of the stack and returning you to that location. You can use the dirs command to display the current directory stack without pushing or popping a new directory. Listing 4 shows how you can use the pushd and popd commands to quickly navigate the directory tree. Listing 4. Using pushd and popd to navigate the directory tree   $ pushd . ~ ~ $ pushd /etc /etc ~ ~ $ pushd /var /var /etc ~ ~ $ pushd /usr/local/bin /usr/local/bin /var /etc ~ ~ $ dirs /usr/local/bin /var /etc ~ ~ $ popd /var /etc ~ ~ $ popd /etc ~ ~ $ popd ~ ~ $ popd   The pushd and popd commands also support parameters to manipulate the directory stack. Using the +n or -n parameter, where n is a number, you can rotate the stack left or right, as shown in Listing 5. Listing 5. Rotating the directory stack   $ dirs /usr/local/bin /var /etc ~ ~ $ pushd +1 /var /etc ~ ~ /usr/local/bin $ pushd -1 ~ /usr/local/bin /var /etc ~   Find large files Need to find out where all your free disk space went? Here are a couple of tools you can use to manage your storage. As shown in Listing 6, the df command shows you the total number of blocks used on each available volume and the percentage of free space. Listing 6. Determining volume usage   $ df Filesystem 512-blocks Used Available Capacity Mounted on /dev/disk0s2 311909984 267275264 44122720 86% / devfs 224 224 0 100% /dev fdesc 2 2 0 100% /dev map -hosts 0 0 0 100% /net map auto_home 0 0 0 100% /home   Want to find the largest files? Use the find command with the -size parameter. Listing 7 shows how to use the find command to find files larger than 10MB. Note that the -size parameter takes a size in kilobytes. Listing 7. Find all files larger than 10MB   $ find / -size +10000k –xdev –exec ls –lh {}\;   Create temporary files without an editor This is a simple one: You need to quickly create a simple temporary file but don't want to fire up your editor. Use the cat command with the > file-redirection operator. As shown in Listing 8, using the cat command without a file name simply echoes anything typed to standard input; the > redirection captures that to the specified file. Note that you must provide the end-of-file character when you're finished typing—typically, Ctrl-D. Listing 8. Quickly create a temporary file   $ cat > my_temp_file.txt This is my temp file text ^D $ cat my_temp_file.txt This is my temp file text   Need to do the same thing but append to an existing file instead of creating a new one? As shown in Listing 9, use the >> operator, instead. The >> file-redirection operator appends to an existing file. Listing 9. Quickly append to a file   $ cat >> my_temp_file.txt More text ^D $ cat my_temp_file.txt This is my temp file text More text   Use the curl command-line utility I can access the Web from the command line? Are you crazy? No, it's just curl! The curl command lets you retrieve data from a server using the HTTP, HTTPS, FTP, FTPS, Gopher, DICT, TELNET, LDAP, or FILE protocols. As shown in Listing 10, I can use the curl command to access the current local conditions of the National Weather Service for my location (Buffalo, NY). When combined with the grep command, I can retrieve the conditions in Buffalo. Use the -s command-line option to suppress curl processing output. Listing 10. Retrieve the current weather conditions with curl   $ curl –s http://www.srh.noaa.gov/data/ALY/RWRALY | grep BUFFALO BUFFALO MOSUNNY 43 22 43 NE13 30.10R   As shown in Listing 11, you can also use the curl command to download HTTP-hosted files. Use the -o parameter to specify where the output is saved. Listing 11. Use curl to download HTTP-hosted files   $ curl -o archive.tar http://www.somesite.com/archive.tar   This is really just a hint of what you can do with curl. You can start exploring a bit more simply by typing man curl at your command prompt to display the complete usage information for the curl command. Make the most of regular expressions Many UNIX commands use regular expressions as arguments. Technically speaking, a regular expression is a string (that is, a sequence of characters composed of letters, numbers, and symbols) that represents a pattern defining zero or more strings. A regular expression uses meta-characters (for example, the asterisk [*] and question mark [?] symbols) to match parts of or whole other strings. A regular expression doesn't have to contain wildcards, but wildcards can make regular expressions useful for searching for patterns and manipulating files. Table 1 shows some basic regular expression sequences. Table 1. Regular expression sequences   Sequence Description Caret (^) Matches the expression at the start of a line, as in ^A Question mark (?) Matches the expression at the end of a line, as in A? Backslash (\) Turns off the special meaning of the next character, as in \^ Brackets ([]) Matches any one of the enclosed characters, as in [aeiou] (Use a hyphen [-] for a range, as in [0-9].) [^ ] Matches any one character except those enclosed in brackets, as in [^0-9] Period (.) Matches a single character of any value except end of line Asterisk (*) Matches zero or more of the preceding characters or expressions \{x,y\} Matches x to y occurrences of the preceding \{x\} Matches exactly x occurrences of the preceding \{x,\} Matches x or more occurrences of the preceding Listing 12 shows some of the basic regular expressions used with the grep command. Listing 12. Using regular expressions with grep   $ # Lists your mail $ grep '^From: ' /usr/mail/$USER $ # Any line with at least one letter $ grep '[a-zA-Z]' search-file.txt $ # Anything not a letter or number $ grep '[^a-zA-Z0-9] search-file.txt $ # Find phone numbers in the form 999-9999 $ grep '[0-9]\{3\}-[0-9]\{4\}' search-file.txt $ # Find lines with exactly one character $ grep '^.$' search-file.txt $ # Find any line that starts with a period "." $ grep '^\.' search-file.txt $ # Find lines that start with a "." and 2 lowercase letters $ grep '^\.[a-z][a-z]' search-file.txt   Many books have been written just about regular expressions. For a more in-depth look at command-line regular expressions, I suggest the developerWorks article, "Speaking UNIX, Part 9: Regular expressions." Determine the current user At times, you may have an administrative script that you want to make sure a certain user has or has not executed. To find out, you can use the whoami command to return the name of the current user. Listing 13 shows the whoami command run on its own; Listing 14 shows an excerpt from a Bash script using whoami to make sure the current user isn't root. Listing 13. Using whoami from the command line   $ whoami John Listing 14. Using whoami in a script   if [ $(whoami) = "root" ] then echo "You cannot run this script as root." exit 1 fi   Process data with awk The awk command always seems to live in the shadows of Perl, but it can be a quick, useful tool for simple command-line-based data manipulation. Listing 15 shows how to get started with the awk command. To get the length of each line in the file text, use the length() function. To see if the string ing is present in the file text, use the index() function, which returns the location of the first occurrence of ing so that you can use it for further string processing. To tokenize (that is, split a line into word-length pieces) a string, use the split() function. Listing 15. Basic awk processing   $ cat text testing the awk command $ awk '{ i = length($0); print i }' text 23 $ awk '{ i = index($0,”ing”); print i}' text 5 $ awk 'BEGIN { i = 1 } { n = split($0,a," "); while (i <= n) {print a[i]; i++;} }' text testing the awk command   Printing specified fields of text file is a simple awk task. In Listing 16, the sales file consists of each salesperson's name followed by a monthly sales figure. You can use the awk command to quickly total the sales for each month. By default, awk treats each comma-separated value as a different field. You use the $n operators to access each individual field. Listing 16. Using awk for data summarization   $cat sales Gene,12,23,7 Dawn,10,25,15 Renee,15,13,18 David,8,21,17 $ awk -F, '{print $1,$2+$3+$4}' sales Gene 42 Dawn 50 Renee 46 David 46   The awk command can be complex and used in a wide variety of situations. To explore the awk command more fully, start with the command man awk in addition to the resources mentioned in the Resources. Conclusion Becoming a command-line wizard takes a bit of practice. It's easy to keep doing things the same way simply because you're used to it. Expanding your command-line resources can provide a big increase in your productivity and propel you toward becoming a UNIX command line wizard!