Go bug my cat!

2015-02-06

Partly related to Francis Irving’s promise to avoid C, and partly related to the drinking at the recent PythonNW social, I wrote a version of /bin/cat in Go.

So far so good. It’s quite short.

The core loop has a curious feature:

for _, f := range args {
	func() {
		in, err := os.Open(f)
		if err != nil {
			fmt.Fprintf(os.Stderr, "%s\n", err)
			os.Exit(2)
		}
		defer in.Close()
		io.Copy(os.Stdout, in)
	}()
}

The curious feature I refer to is that inside the loop I created an anonymous function and call it.

The earlier version of cat.go didn’t do that. And it was buggy. It looked like this:

for _, f := range args {
	in, err := os.Open(f)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
		os.Exit(2)
	}
	defer in.Close()
	io.Copy(os.Stdout, in)
}

The problem with this is the defer. I’m creating one defer per iteration, and they don’t get run until the end of the function. So they just pile up until main returns. This is bad because each defer is going to close a file. If we try and cat 9999 copies of /dev/null we get this::

drj$ ./cat $(yes /dev/null | sed 9999q)
open /dev/null: too many open files

It fails because on Unix there is a limit to the number of simultaneous open files a process can have. It varies from a few dozen to a couple of thousand. When this version of cat opens too many files, it falls over.

In this case we failed because we ran out of a fairly limited resource, Unix file descriptors. But even without that, each defer allocates a small amount of memory (a closure, for example). So defer in loops requires (generally) a little anonymous function wrapper.

I tried rewriting the loop using a lambda and a tail call (see this git branch), but it doesn’t work. The defers still don’t run promptly. (and the tail call is awkward and I had to declare the loop variable on a separate line from the function itself because the scoping isn’t quite right)

2 Responses to “Go bug my cat!”

  1. Gareth Rees Says:

    Looks nice: makes me tempted to have a go (pun intended). I added a couple of comments on error handling (1, 2).

    The defer operation looks interesting, but I wonder how convenient it is in the common case that you have cleanup that you need to do, but only if the function fails. (For example, you’re creating a complex data structure and you want to either create the whole structure, or in the case of failure, none of it.) I guess you’d end up writing groups of statements in threes like this:

    thing, err = do();
    if err != nil { return err; }
    defer func () { if err != nil { undo(thing); } }()

    which I suppose could be worse. (In C you could macroize this kind of boilerplate, but Go doesn’t seem to have macros.)

    • drj11 Says:

      Go has a garbage collector, so generally you don’t need to undo(). But you would for sockets, channels, mutexes, etc.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: