Archive for February, 2015

Go bug my cat!

2015-02-06

Partly related to Francis Irving’s promise to avoid C, and partly related to the drinking at the recent PythonNW social, I wrote a version of /bin/cat in Go.

So far so good. It’s quite short.

The core loop has a curious feature:

for _, f := range args {
	func() {
		in, err := os.Open(f)
		if err != nil {
			fmt.Fprintf(os.Stderr, "%s\n", err)
			os.Exit(2)
		}
		defer in.Close()
		io.Copy(os.Stdout, in)
	}()
}

The curious feature I refer to is that inside the loop I created an anonymous function and call it.

The earlier version of cat.go didn’t do that. And it was buggy. It looked like this:

for _, f := range args {
	in, err := os.Open(f)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
		os.Exit(2)
	}
	defer in.Close()
	io.Copy(os.Stdout, in)
}

The problem with this is the defer. I’m creating one defer per iteration, and they don’t get run until the end of the function. So they just pile up until main returns. This is bad because each defer is going to close a file. If we try and cat 9999 copies of /dev/null we get this::

drj$ ./cat $(yes /dev/null | sed 9999q)
open /dev/null: too many open files

It fails because on Unix there is a limit to the number of simultaneous open files a process can have. It varies from a few dozen to a couple of thousand. When this version of cat opens too many files, it falls over.

In this case we failed because we ran out of a fairly limited resource, Unix file descriptors. But even without that, each defer allocates a small amount of memory (a closure, for example). So defer in loops requires (generally) a little anonymous function wrapper.

I tried rewriting the loop using a lambda and a tail call (see this git branch), but it doesn’t work. The defers still don’t run promptly. (and the tail call is awkward and I had to declare the loop variable on a separate line from the function itself because the scoping isn’t quite right)