Archive for the 'Uncategorized' Category

What you need to be a programmer


You need, above all others:

1) to have dogged persistence in the face of failure.

2) to celebrate the utterly ordinary.

On 1. There is a reason why FAIL Blog is a thing. Failure represents 99% of what programmers do. Programmers do not sit down and write flawless code that works first time. We make mistakes. The compiler complains. The tests fail. It stops when something unexpected happens. Each failure requires the programmer to work out why it failed, fix it, and carry on. I’m talking about failures at every level from the “missing comma” and “expected string not int” type errors, through intermediate errors like using feet instead of metres, or using 100GB of RAM instead of 1GB, to higher level errors like designing a mobile phone app instead of a website.

Some people just cannot do this. They will type in their program, and it will fail to compile or run because of a missing comma or something like that, and they will just stop, and go and make a bacon butty instead. This is perfectly understandable.

Personally, as a programmer, it sometimes feels like it’s not that I want the program to run correctly, it’s that I have a curse where I cannot prevent myself from investigating every last failure and fixing it.

When programming, failure is entirely normal. It takes a certain personality type to be able to cope with this, several times a day.

On 2. The flip side is that when programmers succeed, most of the time we succeed in something entirely ordinary. We have calculated the amount of VAT correctly. We have displayed the user’s email address on the page correctly. Things that are entirely trivial and obviously should Just Work take sufficient effort, and involve overcoming several failures, that just achieving the entirely ordinary seems like it deserves a celebration.

Clearly it is not normal for someone to celebrate the fact that as a result of typing stuff into the CSS file, the dotted border between the header of a table and the next row now displays correctly. So remote are programmers from the rich celebrations that life has to offer that we have to make our own.

There is a horrific irony to all this. Computers only do what they are programmed to do. But it is incredibly difficult to program them (correctly). What they are programmed to do is only an impression of what we intend them to do. In this case it is like trying to take an impression of a fossil with only wet tissue. So, on the one hand everything a computer does it has been programmed to do, on the other it is a monumentally difficult task to get the computer to do anything at all that is useful and correct.

Lovelace grasps this subtle point: “It can do whatever we know how to order it to perform”. It’s the “know how” that’s the tricky bit.

Women of #movember


In the interests of brevity the title says “women” but I mean to include all partners of people growing a mo’ this month.

I am growing a moustache to support a local prostate cancer group.

The reality is that growing a moustache is not hard. I just have to not shave a certain part of my face for a while. Most of the hardship is endured by my partner Philippa who has to put up with bristly kisses for a month.

Support is vital for success. Support for survivors of cancer, and support for sufferers of cancer. Support for people raising money.

I’d like to thank all the partners out there that are supporting their men growing a moustache. I would not be able to do this without the support of my family and friends, and I’m sure others feel similarly.

You can give your support here:



One of the benefits of getting the train from Sheffield to Liverpool is the beautiful views of the Peak District that I don’t pay enough attention to.

Winter is coming, consequently, it’s just the right time to see the sunrise:

copper bronze dashed across the sky
blots and streaks of morning’s blue
a speck of an aeroplane drifted lazily
the buildings of flats and industry were silhouetted
brick chimneys thrust industrially upward
we arrived in Manchester.

Let me start in a way that i


Let me start in a way that i hope you find controversial.

I am against gay marriage. The state should not recognise the marriages of gay couples.

But fear not, I’m a nice liberal Guardian-reading libdem-labour-green swing voter. I am not about to join the UKIP. To expand upon my position:

I am against marriage. The state should not recognise the marriages of couples.

Marriage is a contract between 2 people (and god, if you choose to believe). I don’t see why the state needs to get involved.

There are various rights and responsibilities that come with marriage, but in all cases i think it would be better if these were separated:

– married fathers have more rights (for example, direction of medical care) over their children than unmarried ones. That’s just silly.

– married couples have various tax advantages (rarely, disadvantages) depending of the whims of the current government. It really depends what the policy is here. Is it to encourage people to form lasting bonds, or is it to recognise that people who stay at home to manage a household deserve a tax break to recognise their economic contributions and compensate for their lack of earnings? Two sisters share a home where one has a full-time job and the other looks after the house and does a few hours teaching a week (maybe they co-parent a child). Should they not be able to pool their tax-free allowance? Peoples lives are entangled in all sorts of ways that the “married couple” paradigm does not recognise.

– married couples have access to the divorce courts. Unmarried couples would benefit from the guidance and impartiality of the divorce courts too. Separation is stressful, messy, confusing thing to go through, and of course the divorce court does not wave a magic wand and make it all unicorns and ponies, but i think it is helpful, and it would be helpful to all couples.

– the estate is presumed to pass to the surviving widow when one of a married couple dies.

It’s not that i don’t think the state should be doing these things, in most cases i think they should. It’s that i don’t think these things should be bundled up in Marriage. Sharing your tax-free allowance should be a matter of filing a form with the tax office. Letting your estate pass to another on your death should again, just be a matter of filing a form.

Asserting paternal rights over a child is a lot more messy. Should it matter if you’re not the genetic father of a child, if you’re the one providing the home and investing in the child’s development? Should the mother be allowed to prevent access to a supposed father? I have no idea, but whether a couple is married or not should not matter.

Public recognition. Of course people should be allowed to marry and be married. Public recognition is one of the most important aspects of marriage. But this recognition comes from your peers, not from the state. Couples have been getting married for thousands of years, long before governments came along to record the marriages.

Marriage as a union for life between a man and a woman is a christian thing. Marriage as actually practiced over the last 5000 years or so is much more diverse. We should embrace that diversity (once again). Capturing this in leglisation would be a complicated nightmare.

The simplest thing the state can do is not get involved.

Date formats


Hilary Mason on twitter bemoaned the fact that matplotlib appears to use a floating point number of days to represent datetimes and suggested that “any other standard format” would be better. It is a little bit odd that a Python library uses that format, but it’s presumably because matlab does, and matplotlib is betraying its matlab heritage.

What other standards are there? Well, there’s POSIX time_t which is either an integer or a floating-point type (but actual practice seems to favour integer), and stores the number of seconds since the Unix Epoch, 1970-01-01T00:00:00Z. Not counting leap seconds. Not counting leap seconds is convenient for some calculations, but means there are (recent) times that cannot be represented (namely, any moment during a leap second). That’s not a good representation of time.

There’s another time format in use which is confusingly similar to POSIX’s time_t: it counts time using the number of seconds since the Unix Epoch. Including leap seconds. The opportunities for mistakes in conversion are endless. A problem with this time format is that times in the future (more than a couple of years into the future XX) are ambiguous. Since we don’t yet know if there are going to be leap seconds between now and 2020 we don’t know whether 2020-01-01T00:00:00Z is 1577836800 or a few seconds more or less than that.

An example confusion: “man 2 time” on my Linux box claims that time_t is the number of seconds since Epoch, but “date –date=2013-01-01 +%s” ends in “00” so clearly leap seconds aren’t being accounted for.

That’s twice now that I’ve used ISO 8601: 2013-04-09T06:06:06Z is roughly when I’m writing this paragraph. As a format for representing this times this has obvious benefits and equally obvious drawbacks. The benefit is that it’s pretty clear, and almost human readable (by humans like me). A drawback is that it takes a lot of space: 20 characters as I’ve given it. If you were storing this in a file you could reasonably use the “compact” form: 20130409060606Z, which is 15 characters, or 14 without the ‘Z’. Space misers could encode this in BCD and fit it in 8 bytes, which is no more than a 64-bit time_t.

Another drawback with ISO 8601 is that calculations with times can be a bit trickier. But on the other hand, you’re more likely to get correct answers, and be able to represent all (recent and future) times.

I’ve been working with sea-level and tide data recently and Hilary’s floating point number of days is certainly pretty convenient for that. The calculations are concerned with the average rotation of the earth, and it nicely represents that. Practitioners of the art also seem to like a slightly different time scale which is Julian centuries. Basically the same but scaled by 36525. It’s not unusual for the 0-point to be something like 1900-01-01T00:00:00, or even a midday like 1899-12-31T12:00:00.

What about representing times with sub-second accuracy? When Unix time_t type is an integer type then you’re stuck. ISO 8601 times can be extended with a decimal point and have as many decimal places as you like. So that’s a good representation. Hilary’s floating point number of days is also pretty good, as long as you’re using double precision. If you’re using double precision then even counting Julian centuries is okay, you still get sub-nanosecond precision. JavaScript’s representation is basically a floating point number that is the Unix time_t but in milliseconds not seconds. That too is pretty good.

I think Hilary’s format is actually pretty reasonable. Some calculations (involving counting whole numbers of days between times) are easier, some are tricky (but no trickier than any other format really). All times are representable. There’s some ambiguity about times in the future (that 1200 appointment you make in the year 2020 might actually turn up in your calendar at 12:00:00.5 if there is a leap second, but at least your midnight appointments won’t be on the wrong day). Yes, it’s a litle bit funky that on days with leap seconds the step between consecutive seconds will be different (it will in 1/86359 or 1/86401 instead of 1/86400), but I’m sure you’ll cope.

After a while, you just get used to seeing a new file format, another new date representation. According to the xlrd documentation Excel uses a floating point number of days since 1899-12-30T00:00:00, but since they thought 1900 was a leap year, this is only reliable after 1900-03-01. Unless the Excel file was made on a Mac, in which case the Epoch is 1904-01-01. The TOPEX/Jason data for global mean sea level represents dates as a decimal fraction of years.

The bottom line is: There are all sorts of crazy date formats. Get over it.

I recommend:

1) using a good library.
2) for storage, use ISO 8601
3) for runtime/manipulation use double.

All I need to know to learn R


I’ve been learning R, mostly because it’s been on my list of things to do for ages, and partly because I needed to draw a histogram.

All the tedious stuff about how you get started and how you install things is surprisingly difficult to get from the internets.

So to install on Ubunutu:

sudo apt-get install r-base

Then you run the command R:

$ R

Once you’re in R you can find things out by using Google. Once you’ve found a function you want to use, say sj.test, if it doesn’t seem to be installed, you can install it by noting the library name which is in curly brackets at the top of the man page. {lawstat} in this case. So you then go:

> install.packages("lawstat") # to install it
> library(lawstat) # to use it

(the package installer has a hilariously craptastic interface written in Tcl/Tk)

That’s it. Everything I need to know to learn R. Everything else is just bog standard programming language stuff (though it helps a bit that I learnt a bit of J).

Here’s the histogram:

And the R code is behind this link. The way the function png() implicitly makes hist() write into the PNG file is particularly bletcherous. It has all the elegance of writing JCL for IBM mainframes.

Bosch: the Constructor


We got some second-hand Duplo recently, and washed it. In Bosch, our washing machine. In a pillow case at 30°C on the delicate program. Makes a funny noise but it all came out fine. This was the first time I’d washed Lego in a machine, so it’s nice to know that the suggestion in the Lego FAQ works.

The curious thing was that the Duplo had SELF ASSEMBLED. It went in all in separated bricks. But when it comes out, some of it has stuck together. I found this pretty amazing. Surely if Lego can self assemble in the washing machine then it is a simple step from there to the beginning of life itself.

The variety of forms is interesting. Simple diatoms… in mixed colours:

…and also monochrome:

Simple towers, reminiscent of the things my little nephew makes, when he can be bothered:

Then there comes the forms that are not so easy to characterise. I like to call them monsters:

I’m particularly impressed with Bosch’s creative instincts here. The one at the front is a battleship with effective use of colour. The back left reminds me a bit of a tree. Perhaps some sort of highly coloured baobab tree. And is that a red boot kicking a yellow football on a field of green?

Francis speculates (whimsically) that maybe my washing machine has passed the singularity and is in fact an Artifical Intelligence trying to communicate with me. Obviously it is constrained by only being able to communicate by rearranging whatever I put into the washing machine. And the fact that it is a 4-bit microcontroller attached only to a valve, a heater, and a motor (maybe a temperature sensor too, but I wouldn’t be surprised if there wasn’t one). It’s an intriguing possibility. Perhaps the form on the right of monsters represents a not quite complete utterance, the Duplo bricks not quite bonded properly.

The AI hypothesis raises several questions: How can we test it? Can we distinguish between merely aleatoric arrangements, and intentional ones? What is Bosch trying to say? Is there something I can put in the washing machine that would make it easier for Bosch to communicate? Is Bosch happy?

In praise of ⌘


Control-C is the intr character in Unix.  It sends SIGINT to the currently running process, thereby interrupting it.  Most programs either quit or return to an interactive prompt when they receive this signal.

Control-C is the Copy command in Windows.  It copies the highlighted text (or other graphic objects) to the clipboard.

What do you do when you’re using a terminal emulation program on Windows to connect to a Unix computer?  Does pressing Control-C send a Control-C character to the Unix system, or does it copy the highlighted text to the clipboard?  I have no idea (it’s been a long while since I used a terminal emulator on Windows), but one thing is clear: It’s not clear what the right answer is.

On a Mac Copy is an example of a command to a GUI application.  Keyboard shortcuts are accessed using the Command key.

Command Key

Command Key from wikimedia

This is a stroke of genius.

Now, when using a terminal emulation program on a Mac, it’s completely obvious what happens. Control-C sends a Control-C character to the Unix system, which will interpret it as intr. Command-C copies the highlight text to the clipboard.

Was this luck? Hard to tell, but I think not. The Mac was conceived as an entire product. It therefore seems completely reasonable to design a keyboard just for that product, and completely reasonable to create a new key for doing Mac things. Windows was conceived as an operating system. It had to run on existing hardware.

Why didn’t Apple just stick a Control key on the keyboard and use that? Well, that would have been silly, and difficult to explain. Control had an existing use (on a terminal to send control characters that generally affected either the connection or the movement of the print head) which had nothing to do with GUI commands. Better to name the key that is used for keyboard shortcuts for GUI commands, Command.

Why didn’t Windows do this? Microsoft weren’t in a position where they could dictate what keys appeared on the keyboard. They had to make do with whatever was there. Why Control-C for Copy instead of Alt-C, say? I have no idea. It seems particularly mysterious since, at the time Windows was layered on top of DOS and so was used by people familiar with DOS and in DOS, at least at the command prompt, Control-C meant interrupt, like it does in Unix.

The lucky bit seems to be that Apple ever put a Control key on the keyboard at all; early keyboards didn’t have one, but I guess as soon as you have a modem you can connect to some other computer and that will require a Control key for some things. But basically the Control key on a Mac hung around for 2 decades doing nothing much until OS X comes out and then it’s used in all the traditional Unix ways.

What did Unix do when it got a window system? Well, on X Windows in the mid nineties I don’t remember using a keyboard shortcut for Copy. In fact, I don’t remember using Copy at all. The middle mouse button (most Unix workstations and X servers of the time had mice with 3 or more buttons) would paste the highlighted text into whichever window the mouse was pointing at (note: the highlighted text could be in a different window from the one which receives the pasted text). No separate Copy command, just highlight text in one window, and click middle button to paste it into another window. Cute.

Windows has a Windows button now (and, much to my surprise has had it since Windows 95). Obviously there was no way that Copy could be moved from Control-C to Windows-C because Control-C was already welded into the minds of all Windows users. So the Windows button generally does useless things in Windows.

What about Ubuntu? They could’ve decided that it was reasonable to assume that all keyboards would have the Windows key. They could’ve decided that all GUI shortcut commands would use the Windows key. Freeing up Control-C to mean intr. You would’ve though that for a GUI running on Unix this would be an important consideration. But no, Ubuntu thoughtlessly copies windows and Control-C means Copy. Unless you’re in the terminal emulator, Terminal, in which case it sends a Control-C character and you have to remember to use Shift-Control-C for Copy. And just like Windows, the Windows key is useless in Ubuntu.

It all just seems like a lost opportunity to me.

Not in 2010


Things I predict we won’t be seeing in 2010:

  • Perl 6
  • Latex 3
  • 1:25,000 scale Ordnance Survey vector data made public
  • the hottest November
  • iTunes for Linux
  • reduce in Python 3.x
  • A Green Party MP (you know, the UK parliament, not some cheap plastic Euro knock-off)
  • Natural History Museum: Butterflies


    On Friday popped into the Natural History Museum and went to the Butterfly Jungle. I’m a member, and entry to the for-money exhibitions is free (already paid for). It makes me feel terribly middle class.

    Before entering into the “jungle” (it’s a temporary hut made of out polytunnel) I thought we could go see the insect gallery so we could learn something about butterflies before seeing them. Well, there is no insect gallery, there is the creepy crawlies room. Where’s the long room full of display cabinets crammed with dead insects pinned to neatly labelled pieces of cardboard? Needless to say the creepy crawly room sucks.

    So we sort of wandered about at random. Hey, did you know the toilets have bacteria zapping UV on the hand dryers. Cool. But no interpretation. Not Cool.

    Tania Kovat’s TREE is very good. It’s a slice of a 200 year old oak set into plaster panels in the ceiling; the pieces are arranged more or less how they would have been on the tree, in other words: in the shape of a tree. I find the connexion to Darwin a bit lame. The inspiration is Darwin’s now famous “I think” cladogram from his Transmutation Notebook B (ain’t the Darwin online project great?). The cladogram, you know, looks like a tree. And so does Tania’s TREE. Cunning. TREE is displayed in a rather nice gallery at the top of the splendid staircase in the Central Hall. Behind the statue of Darwin, and between the statues of Hooker and Owen. A holy place.

    In the same gallery is Ida, apparently the world’s most complete fossil primate specimen. She’s a beautiful little bush-baby-like creature, Darwinius masillae. She lived 47 million years ago. Of course, I know the vast majority of species (well over 99%) become extinct, so it is, statistically speaking, unlikely that Ida is our ancestor. Nonetheless it is difficult to dispel the romantic notion that Ida could be our ancestor. Certainly she will have shared a lot in common, looks, behaviour, social grouping, with our actual ancestors. Ida’s cabinet featured something that I think the NHM should have a lot lot more of. A cladogram.

    After wandering past the primate gallery (now quite aging) and the Sequoiadendron giganteum we found the entrance to the Minerals collection. I didn’t actually know the NHM did rocks. And this is awesome. A big gallery full of oak cabinets (original 1881!), stuffed full of… rocks! We didn’t want to spend much time here (we were getting hungry), but I thought it would be interesting to see what the NHM had to say about the alexandrite effect and birefringence. It was a simple pleasure to use the alphabetic mineral index to find the cabinet displaying alexandrite.

    Alexandrite appears to be different colours under different lighting conditions. One colour under natural sunlight, and a different colour under incandescent light. I was slightly disappointed to find that cabinet didn’t have a button to press to illuminate the alexandrite with different lights. Oh well. I suppose every mineral is special in its own way, so I can’t expect every one to have a cute interpretation. Of some local interest to me was spotting the enormous Blue John specimen, about as big as my chest. Blue John is a fluorite variety local to Castleton. Of course, I’ve seen far better examples in the shops in Castleton.

    I knew quartz was a birefringent material, so I popped over to the quartz cabinet. No birefringence here. As we were ambling out of the room, I luckily found a fine quartz crystal ball on display in the jewellery cabinet next to a rather fine jade box on loan from the Queen. Gazing into the crystal ball gives the birefringent double image effect (this is deliberate, there is an interpretation sign to explain the effect). Nice, but I think Wikipedia’s image is more impressive.

    After lunch and a quick trip round the wildlife garden (impressive use of a small urban space, and I expect it to keep improving; didn’t see the foxes though) we did eventually make it to the Butterfly Jungle.

    Which I thought was a bit disappointing. However, there’s something intrinsically delightful about having lots of butterflies flapping about, and it hard not to enjoy that rather pleasant experience. And they are pretty to look at. As for science though, there was precious little to be found (not none, but not a great deal). It wasn’t all butterflies, there was an amusing collection of slightly exotic creaturees in glass cages. Giant african millipede, death’s head cockroach, Charlie the Iguana iguana (who I last saw in the Darwin exhibition!); that sort of thing. And a kiddies playground. Which looked quite good, but no use to me.

    The Natural History Museum is such a large museum and with so much on display, I find that it’s impossible to do anything but see a small sample of it in any of visit. I’ve been three times recently and I’ve still only seen a small fraction of what it has to offer. There are still things to discover in the Central Hall: I was pleased to see a Glyptodon that I had missed on my previous visits.

    I did learn one thing in the Butterfly Jungle. Butterflies taste with their feet.


    Get every new post delivered to your Inbox.

    Join 25 other followers