Shell Tips

Introduction

The following are a number of tips and tricks for the unix shell environment. Shell examples assume a non-csh-based shell, such as bash or zsh. Consult the manual for the commands in question if you see errors, as your version of the tool may be different from the one shown here.

Articles

Longer discussions that warrant own page due to length or for reference purposes.

Tips & Tricks

Some random shell tips and tricks.

Redirection.

Redirection can take place pretty much anywhere on the command line. Consider the following.

$ echo a b >c
$ echo >c a b
$ >c echo a b

Pipe to command arguments.

I use xargs(1) frequently to convert output from something (file, or another program) to arguments to another command. For instance, to commit only modified files in a cvs sandbox where there may be conflicted, new, or other troublesome files mixed in, use the following.

$ cvs up | perl -nle 'print if s/M //' | xargs cvs ci

Depending on the editor, one can use concept above to open certain files for editing, for example, files in a cvs sandbox that have conflicts.

The bbedit utility is part of BBEdit for Mac OS X, and avoids terminal issues by sending the files to the BBEdit application. Using emacs in server/client mode may avoid this problem for emacs. The alternative is to use backticks to make the files available as arguments to the program, instead of feeding them in through xargs.

$ vi `cvs up | perl -nle 'print if s/C //'`

xargs can be chained with other programs. For instance, one may want to find perl scripts containing the text While and do something with them.

$ find . -name '*.pl' \ | xargs fgrep -l While \ | xargs perl -i -ple 's/While/while/g'

xargs will fail or do the wrong thing if passed filenames contain spaces. This is common on filesystems that traditionally have allowed spaces in filenames (Mac OS), or where file trees have been uploaded to unix from other platforms. If using find/xargs pairs, the spaces-in-filenames problems can be avoided as follows.

$ find . -type f -print0 | xargs -0 echo

Stubborn Files

Dealing with files that have odd characters in their names can often be a chore on unix, as one cannot type in the names in question. One could use a graphical file manager tool, but I find those cumbersome, ill suited to dealing with large numbers of files, and usually not installed on server systems.

To simply delete the bad filenames, there are a few options.

Delete all interactively.

$ \rm -i *

Say “yes” for each of the bad filenames, no otherwise. The backslash is to prevent alias expansion on rm, which might turn off interactive mode. Quick solution for cases where the glob can get you close enough, usually with limited numbers of files to work with.

Find inode and delete by that.

Each file on a unix filesystem has a inode number associated with it; knowing the inode number of the bad file allows us to search for and delete it.

$ ls -i *
615383 foo
$ find . -inum 615383 -exec rm {} \;

Large numbers of files.

If there are large numbers of files with wacky characters in their filenames, something more powerful than the shell is usually required to filter out the files in question. For instance, to list the inode number of files in the current directory with non-printable characters in their names, use perl.

$ ls -i | perl -nle 'print if /[[:^print:]]/' \ | while read inum name; do echo $inum; done

For situations where the mangled filenames are in deep directory trees, or where the mangling is consistent (uploaded filenames from a DNA sequencer come to mind), use File::Find and write a standalone script.

Fun with SSH

Commands can be run over ssh(1), though how the shell handles more complex commands can cause problems.

client$ ssh example.org hostname
server.example.org
client$ ssh example.org sleep 3 && hostname
client.example.org

The && is handled by the local shell, not the remote server. Quoting can fix the problem.

client$ ssh example.org 'sleep 3 && hostname'
server.example.org

Clobbering a File

There are several ways one can empty the contents of an existing file without removing and touch(1)-ing the file in question. Using echo(1) is not portable, as some systems do not support the -n flag, such as Digital UNIX without CMD_ENV=bsd set. The use of the shell null operator : is a clever way, and saves typing.

$ cat /dev/null >file
$ echo -n >file
$ : >file

Grepping for processes

A problematic feature of grepping for processes is that the grep can sometimes catch itself, a boundary condition best avoided, as killing the grep may mess something else up. To avoid this problem, compose the regular expression in such a way as to prevent grep from matching its arguments.

$ ps wwo pid,command | grep ss[h]

The above command uses a character class in the regular expression, and avoids being caught itself, as ss[h] (which looks for the literal string ssh) cannot match itself (ss[h]) in the command field of the process table.

Shell Tips

Introduction

Articles

Tips & Tricks

Related