Archive for the 'unix' Category

Unix command line pebbles


I always find that “sitting next to someone new” is an interesting and revealing experience. I learn new things, they learn new things. Over the years I’ve accumulated a bunch of tricks about using the command line, using vi, and even using emacs. The thing I’m talking about are things can’t really be learned in a formal context, they are too small and there are too many of them. Yet collectively they form part of the fabric of what it means to be a vi user or a shell programmer or whatever. Picking them up and passing them on by sitting next to someone is the only way I know to spread them. Well, maybe this blog post is a start.

Here’s a few I’ve recently picked up or passed on:

cd -

It changes directory to the last directory that you changed from. It’s /bin/bash not /bin/sh (so I personally would avoid writing it in scripts). I think someone alleged that this was undocumented, but in fact I recently checked and it is in fact documented in the builtins section of the /bin/bash man page, which is huge.

It’s useful when you want to cd to some directory to run a command, but then cd back.

Line 25 of the runlocal script in ScraperWiki used to do just that in order to start a service from within a different current directory:

cd ../services/scriptmgr
node ./scriptmgr.js &
cd -

Because I don’t like writing bash-specific scripts, I changed this to use a subshell, with round brackets:

    cd ../services/scriptmgr
    node ./scriptmgr.js &

A subshell is used to run the code between the round brackets, and it is just a forked copy of the shell. It has exactly the same environment (environment variables, open files, and so on), but is in another process. Since the Current Working Directory is specific to the process, changing it in the subshell has no effect on the outer shell, which carries on unaffected after the subshell has returned. So that’s something I’ve passed on recently.

cp foo.cnf{-inhg,}

This is the same as cp foo.cnf-inhg foo.cnf. It copies a file by removing the -inhg extension from its name. ScraperWiki stores some config files in Mercurial (the -inhg versions), and they are copied so they can be edited locally (the versions without the -inhg suffix). I’m sure this has all sorts of evil uses.

sudo !!

Of the things I’ve picked up recently, this is my favourite. I like how some of these tips are a combination of things that I already knew individually but hadn’t thought to combine. Of course I know that sudo thing runs thing as root, and I did once know that !! is the /bin/csh mechanism (stolen by /bin/bash) for executing the previously typed command. In combination, sudo !! runs the previously typed command as root. Which is perfect for when you typed pip install thinginator instead of sudo pip install thinginator.


The favourite thing I seem to have spread seems to be Ctrl-R. (in /bin/bash) you press Ctrl-R, start typing, and bash will search for what you type in your command history; press Return to execute the command or Ctrl-R to find the next older match. Press Ctrl-C if you didn’t really want to do that.


One of the @pysheff crowd for sudo !!; can’t remember whether it was @davbo or @dchetwynd.

Ross Jones for cd –;

Bitbucket user mammadori “{-ext,}”.

If you enjoyed this post, you may also like Mark Dominus’ semi-rant on the craptastic way to do calculation in shell scripts. Huh, now I feel compelled to that Mark gets it wrong about GNU Shell; arithmetic expansion is not a GNU innovation, it comes from POSIX, having been slightly mutated from an earlier Korn Shell feature. And if he had comments I would’ve said that on his “blag”.

Shell programming: The fu example


Many years ago, back when I thought spending time configuring my Unix environment was a good idea, I wrote a script that I call fu. It helps manage long pathnames and is particularly useful with our tendency to have deep directory structures in perforce:

$ pwd
$ fu master
$ cd `fu master`
$ pwd

As you can see, fu searches up the directory hierarchy (towards root) until it finds a match for its argument.

I think its code provides some interesting points of shell programming that are not always stressed:

# $Header: //depot/home/drj/master/prj/uxutils/fu#3 $
# fu - find up
# finds a file who's name is / where x is specified on the
# command line, and  is some prefix (dirname) of the current
# working directory.



while :
  case "X$d" in
    */) d=`echo "$d"|sed '$s/.$//'`;;
  if test -r "$d/$x"
    echo "$d/$x"
    exit 0
  n=`dirname "$d"`
  if test "X$d" = "X$n"
    exit 4
exit 2

The first things to point out is all the double quotes around variable expansions, like this «”$d”». That’s done so that the variable expands into a single word even when it contains a space. This is almost always what you want. Consider a simple example like «ls $d», if d is the string «foo bar» (which contains a space) then this will ask ls to list the two files «foo» and «bar», which is probably not what you wanted. Of course people that put spaces in their filenames, then pass those names as arguments to Unix programs, then expect it to work, are insane (are you reading this Apple? «Library/Application Support/»). But we do what we can. So unless there are very good reasons why not, every variable expansion gets double quotes around it.

The line that invokes sed, «d=`echo “$d”|sed ‘$s/.$//’`», is pretty typical of string manipulation in shell: tedious, ugly, and sometimes obscure. All it’s doing is stripping a trailing «/» from d, but look how awkward it is. It invokes an external program and probably forks another shell; it could easily be a million times slower than the equivalent code in more traditional compiled language like C or Lisp. You don’t have to do very much of this sort of fiddly manipulation before it becomes very sensible to use a proper language like Python.

«test -r» is evidence of how old the script is (I think). -r tests whether a file is readable, whereas all I care about is existence; clearly I should be using «test -e», but I suspect that at the time I originally wrote the script -e was either not standard or not implemented widely enough. These days I should probably change it.

Those initial Xs in «if test “X$d” = “X$n”» are kind of curious. They’re there to prevent the kind of nonsense that happens if d happens to start with a hyphen and therefore confuse test. If d happened to be «-x» for example, then we would have test -x = something which might confuse test into thinking that it should be executing the -x test (test for an executable file). The version of test on OS X doesn’t suffer this problem, but I’m pretty sure that earlier ones did. I’m pretty sure that the X in «case “X$d”» is there for the same reason, but that’s a bogus reason because case doesn’t suffer this problem.

Hmm. Well, perhaps it would made for some interesting points of shell programming if two of my circumlocutions weren’t made obsolete by progress in Unix utility implementations.

Awesome Shell Power: stupid z trick


Say you want to move all the files in the current directory down into a new directory called “awesome”.

Instead of:

mkdir awesome
mv * awesome

which is likely to complain: “mv: rename awesome to awesome/awesome: Invalid argument” (though it does in fact work), you might try:

mkdir z
mv *
mv z awesome

This “works” when the newly created directory, called “z”, comes at the end of the list that “*” expands to. You weren’t in the habit of creating files beginning with z were you?

Awesome Shell Power: yes ” | head | nl -ba


Often when programming in shell you want a sequence of numbers. bash has quirky for syntax for doing it, but I’m not really interested in that. I want something portable.

«yes '' | head | nl -ba» will generate a sequence of numbers from 1 to 10. 1 to N can be achieved by specifying an argument to head: «yes '' | head -7 | nl -ba», or, as I never tire of telling people, using sed instead of head: «yes '' | sed 7q | nl -ba». The sed version is one character shorter.

As long as we’ve invoked sed we may as well read the manual to get: «yes | sed -n '=;7q'». Note that as sed is not actually processing the input lines, just counting them, it doesn’t matter what their contents are, so we no longer need an argument to yes.

It turns out that yes is not a standard part of the Unix utilities. So much for portability. Whilst we could make an adequate simulation of yes with «while : ; do echo y ; done», it’s much more amusing to use dd:

$ dd < /dev/zero ibs=1 cbs=1 count=7 conv=unblock | sed -n =
7+0 records in
0+1 records out
14 bytes transferred in 0.000060 secs (233945 bytes/sec)

I love dd for its obscurity, its parody of the DD statement in JCL, and the way no-one seems to know what it is for any more. There are some stderr turds left by dd which we can eliminate with a quick 2>&-: «dd < /dev/zero 2>&- ibs=1 cbs=1 count=7 conv=unblock | sed -n =».

This version of the command is improved in many ways: “count=7” is practically self-documenting; we no longer need quotes around the sed program (though the way a bare = appears at the end does seem like a bit of a non sequitur, especially given all the other = that appear in the dd command that have a completely different meaning). However, it’s unsatisfying in many other ways. It uses /dev/zero which isn’t guaranteed to exist (even on conformant Unix OSes). It squirts binary data (a sequence of alternating NUL and LF characters) into sed which isn’t obliged to behave on such input. It’s long and silly.

Much better to just generate the sequence directly using something approaching a real programming language:

awk 'BEGIN{ for(i=1;i<=7;++i) print i}'

Even those who don’t know the mighty awk can probably work out what it does. It’s portable (some now truly ancient, that is SunOS 4.x, systems might need an extra < /dev/null); it’s clear; it works.

Of course on systems with jot installed you can just go jot 7. That includes FreeBSD and OS X. It turns out that jot is named after APL’s ɩ, iota, which is spelt i. in J.

Awesome Shell Power: 2>&1 | tee …


Collect the stdout and stderr of a command into a single file and see the output interactively:

some command 2>&1 | tee err

I use this with make so often that I have a script called mk:

make "$@" 2>&1 | tee err

(and don’t forget, we always use "$@", never $*)

The output file is named err but would probably be better named ofile (suggested by Nick B in comments). I use err in deference to 15 years of my personal Unix tradition.

The Order of Redirections

Now let’s take pipe out of the equation for a moment and consider redirections alone. On my Mac OS X laptop the command ls -d /eta /etc produces the following output:

ls: /eta: No such file or directory

The “No such file or directory” message appears on stderr and “/etc” appears on stdout.

You can prove that to yourself by closing one or the other file descriptor. Play around with ls -d /eta /etc >&-, and ls -d /eta /etc 2>&- and convince yourself that this is so.

Shell redirections are processed from left to right; we can get the stderr of a command to appear on its stdout and get rid of its stdout entirely:

$ ls -d /eta /etc 2>&1 1>&-
ls: /eta: No such file or directory

You might like to think about why this works. fd 2 is redirected to fd 1, meaning output to fd 2 goes to the same place as fd 1 at the time that the redirection was made. Then fd 1 is closed; this doesn’t affect the previous redirection of fd 2.

Doing it the other way round, we get this:

$ ls -d /eta /etc 1>&- 2>&1
-bash: 1: Bad file descriptor

That’s because by the time that it comes to redirect fd 2 to fd 1, fd 1 has been closed and can’t be used for a redirect.

How can we tell that 2>&1 1>&- sends what normally appears on stderr to stdout? Does this convince you:

$ ( ls -d /eta /etc 2>&1 1>&- ) | sed 's/^/*** /'
*** ls: /eta: No such file or directory

By borrowing an extra fd you can even swap stderr and stdout over:

$ ls -d /eta /etc 3>&2 2>&1 1>&3 3>&- | sed 's/^/*** /'
*** ls: /eta: No such file or directory

Notice that the error message from ls has gone through the pipe and been transformed by sed, so it must have been sent to stdout. The normal output of ls, “/etc”, has not gone through the pipe, so must’ve been sent to stderr.

Awesome Shell Power: */.


I probably know lots of tips and tricks about using /bin/sh that are not widely enough known about.

Today’s shell tip: */.

It’s a shell pattern that expands to all the sub-directories of the current directory.

If you want to list all the names of the sub-directories in the current directory it’s a lot easier to type echo */. than it is to type find . -name . -o -type d -print -prune which is almost equivalent.

ls -ld */.