My Enbridge pipeline


In April 2012 the town of Bella Bella, mostly Heiltsuk First Nation, were expecting to meet representatives of the Crown Canadian government to discuss Enbridge’s proposed Northern Gateway pipeline. This was to be a pipeline carrying diluted bitumen to a terminal in British Columbia. Where the Heiltsuk fish. The Heiltsuk are a community intimately connected to their land. The coast, the sea, the cedar and fir forest. And the sockeye salmon. The sockeye salmon is an emblematic species and it is connected to its regional ecology through a network of interactions, eating, being eaten, and of course the migration from freshwater to sea and back before spawning in the natal fresh waters. The sockeye salmon reaches out and connects all the elements together.

sockeye salmon migration

sockeye salmon migration (credit: Felex Liu)

The pipeline threatens all of this. Jess Housty, poet, librarian, and young leader, recalls to Naomi Klein their stance: “We were prepared to stand up with dignity and integrity to be witness for the lands and waters that sustained our ancestors—that sustain us”.

As it happened, the government delayed the hearings and moved them so that Jess had to travel for a day to make it to the hearing. Where they said “I respectfully disagree with the notion that there is any compensation to be made for the loss of our identity, for the loss of our right to be Heiltsuk.”

These words, as I was reading them in Naomi Klein’s «This Changes Everything», struck me. Powerfully.

The Heiltsuk are so intimately connected with their landscape and their ecosystem that a pipeline is not just a threat to their food or their way of life, it is a threat to their very identity. The pipeline threatens their identity and it is not surprising that they dig in, resist, and return the assault.

We all have identities that can be threatened.

I am from the internet. My ecosystem has components like computers, and online interactions, and networks, and data, and encryption. Encryption is a keystone species. Take it away and the network around it will collapse. Maybe not instantly, maybe gradually. Shopping will be less secure. My spending patterns just a little bit different. Our conversation less vibrant. Eventually the network will collapse, leaving nothing but sterile scorched earth.

When you attack encryption you attack my identity. And there is no compensation to be made.

Free discourse


Inside your own head you have the freedom to think whatever you like. But we are social animals, discourse is human. We should be free to voice our thoughts to one another.


We are also tool makers. It’s natural that we extend discourse via our tools. First a slit drum, then WhatsApp. I don’t think our tools should change our expectations about discourse. I should be free to choose my conversational partners.

And by that, I mean you and I. I don’t mean, you and I, and the app maker, and the telecoms company, and the government, and the government’s international allies, and the government’s well-funded international enemies who are able to steal the backdoor key.

Discourse is part of our identity and our agency. Without free discourse our agency erodes (you can’t talk to your friends about X), when our agency erodes our identity erodes.

That’s why I support strong end-to-end encryption.

(image from wikipedia:

How I Learnt Python


Judging from the people at PyCon UK that I spoke to, there was quite a lot of interest in learning Python. This is all good and to be expected. Not only are conferences a good place for the Python-curious, this conference had the Trans*Code, DjangoGirls, and the education track, all aimed at people who had never programmed in Python before.

ZX81 Basic Manual

ZX81 Basic Manual

Several times I got asked, “So, how did you learn Python”. How I learnt Python was that I learnt Sinclair BASIC from Steve Vickers’ ZX81 BASIC Programming when I was about 10 years old. A combination of a wobbly RAM pack and unreliable tape means you get good at typing programs in. Then I learnt a bit of Zilog Z80 machine code and wrote my own assemblers, and dabbled in FORTH. At sixth form I got an Atari ST, and I learnt Motorola 68000 assembler and started to dabble in C, Lisp (XLISP), more FORTH, and extensible text editors (MicroEMACS). At university I found C, more Lisps (Cambridge, Acorn, Common), more extensible text editors (the wonderfully charming and parochial ZED and E), scripting languages (WREN, lol), and a bit of ML. Towards the end of university I started to learn about Unix and ed, sed, vi, emacs, and awk. After that I got a job as a C programmer and learnt a lot more C and bit about the SPARC, UltraSPARC, MIPS, and ALPHA architectures. As well as more Unixes, and a bit about Windows NT, and Dylan. Java appeared on the scene (and I did write one small program in it). After that I learnt Lua. And then, eventually, Python. As you might imagine, at this point, learning Python is rather easy. There is almost nothing new in the language except for the indentation.

So my answer to the question is not very useful. Most people who ask the question want to know “how should they learn Python?”, and how I learnt Python is clearly not how they should learn Python. I’m not really sure I have a good answer, even though I’ve been involved in teaching it (at Software Carpentry, and with John Pinner at a PyCon). Python has an official tutorial, and I used that a long time ago and thought it was not bad. I’ve heard some fairly lukewarm responses from other beginners though. Software Carpentry is good, but not very frequent unless you want to travel a long way (and it is somewhat targeted at the scientific post-doc). I’ve never tried any of the MOOCs. I’ve seen someone go through many Python koans, but it didn’t seem to help them learn Python any faster than any other way. DjangoGirls had a very positive response at PyCon UK, and not only teaches Python from scratch but also Django, so you can make a useful website.

A buddy will help. A concrete goal will help, I love to quote Nardi:

“People are likely to be better at learning and using computer languages that closely match their interests”

Bonnie Nardi, “A Small Matter of Programming”

So if you like cycling write an app to calculate your favourite gear ratios, if you like climate science, try recreating the global historical temperature change.

If you haven’t learnt Python yet, how are you planning to? If you’re learning Python, how are you doing it? If you know Python, how did you learn it?

The Diverse People of PyCon UK


I met the Irish, the Greeks. I met the Django Girls, the Trans*Coders, the Posters. I met the teachers, the testers, the gamers. I met old friends, I tended acquaintances hoping to grow them into friendships, and I made new friends.

Diversity is good.


I don’t know if you’ve noticed, but we have a diversity problem. By “we” I mean the programming community. One could spend an entire career and meet only white middle class straight cis men. If you have the opportunity, try and avoid that.

To quote Rachel Evans summarising the PyCon UK attendance: 300 delegates on the main track, 55 Django Girls and TransCode, 30 scientists, 40 teachers, and 100 children; 20% women.

I’m glad someone is watching (and I would encourage the newly formed PyCon UK Committee to appoint a diversity officer, to watch on a formal basis). And I’m pretty sure that Rachel is right, this is better than many parts of the tech sector. My own experience is that professional interactions are > 95% with other men. And of course gender is just one small aspect of diversity, and male/female is just one aspect of gender. So, the non-white, the trans, the gay, all under-represented.

That is why I agree with Rachel that there is still room for improvement, and that Belinda Parmar is right to point out that it’s another dude fest. I was in that hall and it certainly had more “dudes” than the morning before when the ranks of dudes were swelled by our allies Trans*Code, Django Girls, and teachers. On a positive note, the dudes have turned up to see one the the Django Girl organisers, @helenst, present her experiences on Managing Mocks.

Daniele Procida’s keynote was about how in the Open Source world power is left lying around for the taking. Whom you pay attention to is a small example of that power. Stop going to talks by white middle class straight cis men, stop reading my blog, and start paying attention to the diversity that your community already has. Foster it. Fund it. And don’t worry, the poor downtrodden white cis man will still have plenty of opportunity to give talks and plenty of people listening to them.

I think PyCon UK is doing great. I went to see the science track, the education track, Trans*code, DjangoGirls. And once I’d done that, the so called “main” conference didn’t seem so main any more.

Talking PyCon UK


I’m always so grateful for the speakers. For without speakers there would be no conference, and without the conference there would be no people.

I remember a few years ago at a EuroPython in Birmingham, in the morning I met someone for whom it was their first conference. Towards the end of the first day I spotted them again and they looked very tired, I asked them how many talks this person had been to and they replied “all of them”. “Beginner mistake!” I replied, and explained that if you went to all the talks that you could then you would be very tired, your brain would fill up and overflow, and in any case “the corridor track” was where it was at.

This year I think I tried for about 50:50 talks and “corridor track”. If you’re not familiar with the term “corridor track” it means all the talking and conferring that goes on in the corridor, outside the scheduled talks. If you think about it, it’s really the only place you can properly confer with someone, which is the whole point of a conference. But this blog article is about the talks.

I went to the @ntoll and @teknoteacher show who were cajoling and encouraging the teachers in the education track, being generally enthusiastic about all things Python and all things education, and introducing developers to teachers. In a similar vein I went to (stumbled into would perhaps be more accurate) MissPhilbin‘s “colouring in” class. It was actually a practical hands on workshop using the Pi Foundation’s new Sense Hat. I was impressed at the mix of skills needed (bit of design and colouring in of pixel art, bit of colour theory, bit of programming, and interaction design). More of that please.

Daniele Procida’s keynote “All I Want Is Power” was very smoothly delivered and mixed light moments with really serious commentary. In an open source world the power is knowledge and doing (the “do”-ocracy, remember?), and it’s lying around for the taking. So we should take it and begin the revolution. Something like that anyway.

I might not have agreed with everything Owen Campbell said in his “Leadership of Technical Teams” talk, but the subject is important. I don’t think programmers as a whole talk enough about people skills, or indeed all the other things we do that aren’t programming. So this was a welcome talk, and certainly had good points to make, and a useful diagram for framing expertise (page 17 of the slides). In the area of leading technical teams, Owen has much more expertise and experience than me, so it doesn’t matter that I disagree, he would ignore me anyway. :)

I finished the day with the Safe and Fiona show. Talking about their experiences of making cross platform (meaning mobile platform these days) games and finding and building the tools and frameworks to make them. The quest ends with using Kivy and building Myrmidon. It was a talk in two parts, Safe delivering part one and Fiona delivering part two, and I thought that worked quite well.

Poster Session

Another early start on Saturday (I was driving from Sheffield), where after breakfast I watched Simon Sheridan’s talk, “Landing on a comet”. I was a little late, so if Simon started by explaining what his Ptolemy experiment had to do with Python, then I missed it. In any case, no matter about the Python connection, the talk was really interesting. Science, comets, serendipitous mishaps, all good fun. Because I was slightly late, I watched this from the overflow room, which I thought worked rather well. The audio and video feeds from the main room were brought into a side room where you could watch on the projector. I liked it.

Zeth is one of those people who seems to have only a single name. Just Zeth. His talk on JSON and what to do when you want to add a bit of structure to JSON was somewhat misleadingly title “JSON and the daughters of Pelias”. I didn’t get the reference to the ancient greek Jason, husband of Medea, until Zeth had practically spelled it out for me. Adding type systems to JSON is always good fun.

I wandered out to get a cup of tea, then realised I was too late to rejoin to see @helenst‘s talk on using Python mocks. Due to an all too common problem getting the laptop to talk to the main projector, the talk was late starting, so I did in fact manage to sneak in. Despite having to accelerate the talk to make it 10 minutes shorter, Helen’s talk was well delivered and very practical. I’ve used mocks a bit in the past and experience many of the same situations and experiences that Helen had, so I found a lot to sympathise with.

Just before lunch Gemma Hentsch revealed why she has an unhealthy love of tests. I think all programmers are a little bit obsessive about something, so it’s nice to see this come out in a talk.

Thanks to @morty_uk I found an almost secret staircase that led to the poster session. I think this was new this year and it was most welcome.

After lunch I went to @flubdevork‘s talk of packaging with Conda. Anaconda is pretty popular in scientific circles and uses Conda. So I think this is something that I’m going to be seeing more of. I also, between the two lightning talk sessions, managed to see Tim Golden talk, very rapidly, on his experiences using pgzero and Raspberry Pi to teach a small group of young teenagers.

Ah the lightning talks. At my first conference I had no idea what lightning talks were, now I see them as one of the best parts of the conference. People stand on stage in front of the big hall and talk for 5 minutes. Lightning Talk Man comperes and provides witty German jokes. I think we collectively must be getting better at lightning talks, because they were mostly pretty slick. It was especially exciting to see the kids do their own lightning talks earlier in the afternoon. Children just seem to have an infectious enthusiasm when they get into something, and it is really encouraging to see them getting coding.

In between all this, I did manage to chat to a few people and share in the communion of tea and cake.

Return to PyCon UK


PyCon UK 2015 is the UK’s Python conference. Held again in Coventry. I didn’t go last year, in fact I have missed a few years and this has been my first PyCon for a while.

This year we are without John Pinner. I was sad to hear of his death earlier this year. John brought me to the Python community. When I turned up to the first PyCon UK way back in 2007, 10 minutes before I was due to give a talk, John made me immediately feel welcome. The efforts of John Pinner and The Committee have always made PyCon UK a rewarding and invigorating experience. I’m drinking from the very splendid conference mug, which has a lovely quote from John Pinner back in 2006, before the first PyCon UK:

What we really need to do is set up a UK Python conference.

Thanks John.

PyCon UK 2015

For me the conference is about the community: the people attending, the committee, the “do”-ocracy spirit. Conferences can be stressful on a personal level what with new places, hundreds of people you don’t know, strange food. It didn’t take me long before I was relaxing a bit and chatting to new friends. I think the Python community is very welcoming, I know I’m not the only one who thinks that, and I hope everyone attending is able to share that spirit.

It was pleasing to see old friends (all a bit older!), and very pleasing to make new ones, particularly the Python Ireland people enthusing about the upcoming PyCon Ireland. Yet again there was not enough time to meet all the interesting people that I wanted to, but I think that will always be true.

One of the more endearing memories for me was when the daughter of Zeth, in a moment of inspired parenting, manged to get the microphone in the plenary session and was merrily shouting “MAC ADAPTER! MAC ADAPTER!”, clearly pleased to the moon and back to get the attention of 400 or so PyCon UK attendees :).

Thanks to the committee and the volunteers, past and future!

Dictionary ordering and floating point


This is an expanded version of an earlier tweet of mine.

Consider the following Python program:

    d = {k:k for k in 'bfg'}

The output of this program changes from run to run:

    drj$ ./ 
    {'g': 'g', 'f': 'f', 'b': 'b'}
    drj$ ./ 
    {'b': 'b', 'g': 'g', 'f': 'f'}

Dictionaries in Python are implemented using hash tables, so when iterating over a dictionary, the order of the keys is not generally predictable.

You might have known that it wasn’t predictable, but you might not have known that it can change from one Python process to the next. This is a good thing. It avoids a denial-of-service attack that is cross language and cross framework. But don’t worry about that, it’s been fixed since Python 3.3, released over 2 years ago now.

Recently we (at ScraperWiki) came across this in the context of floating point addition. What if the order of a dictionary determines the order of a summation? This program adds up the values of a dictionary:

    d = dict(b=0.2, f=0.6, g=0.7)

It doesn’t always produce the same answer:

    drj$ ./ 
    drj$ ./ 

This is obviously a very smaller difference, but as it happened we were dynamically generating a web page that showed rounded results. 1.4999999999999998 rounds to 1, 1.5 rounds to 2. You could reload the web page and the value shown would sometimes flip from 1 to 2, and vice versa (well, these aren’t the real numbers, but you get the idea).

There are a few things worth pointing out here. As a convenience instead of writing “3602879701896397 / 18014398509481984” I’ve written “0.2”, and similarly “5404319552844595 / 9007199254740992” is written “0.6”, and “3152519739159347 / 4503599627370496” is written “0.7”.

The second program is just calling sum(), so in this case I don’t get to pick the order of adding up for two reasons: 1) The order of the list that is input to sum() is not chosen by me; 2) The order in which sum() sums things might be different anyway (this turns out to be a surprisingly subtle subject, but I don’t want to get into math.fsum).

Tweets like this: “I don’t care if float addition is associative or not” miss the point I was making, which is that floating point addition is not associative (meaning that (0.2 ⟡ 0.6) ⟡ 0.7) != 0.2 ⟡ (0.6 ⟡ 0.7) where ⟡ means floating point addition). Floating point arithmetic is an abstraction and sometimes you need to understand the abstraction and what lies beneath in order to debug your webpage.

In this case we fixed the problem by avoiding floating point until the very last step. The numbers we were adding were all of the form p/20 (where p is an integer). So doing the adding up in integers and then doing a single division at the end means that we get the exact answer (or the nearest representable floating point number when the exact answer is not representable).

What you need to be a programmer


You need, above all others:

1) to have dogged persistence in the face of failure.

2) to celebrate the utterly ordinary.

On 1. There is a reason why FAIL Blog is a thing. Failure represents 99% of what programmers do. Programmers do not sit down and write flawless code that works first time. We make mistakes. The compiler complains. The tests fail. It stops when something unexpected happens. Each failure requires the programmer to work out why it failed, fix it, and carry on. I’m talking about failures at every level from the “missing comma” and “expected string not int” type errors, through intermediate errors like using feet instead of metres, or using 100GB of RAM instead of 1GB, to higher level errors like designing a mobile phone app instead of a website.

Some people just cannot do this. They will type in their program, and it will fail to compile or run because of a missing comma or something like that, and they will just stop, and go and make a bacon butty instead. This is perfectly understandable.

Personally, as a programmer, it sometimes feels like it’s not that I want the program to run correctly, it’s that I have a curse where I cannot prevent myself from investigating every last failure and fixing it.

When programming, failure is entirely normal. It takes a certain personality type to be able to cope with this, several times a day.

On 2. The flip side is that when programmers succeed, most of the time we succeed in something entirely ordinary. We have calculated the amount of VAT correctly. We have displayed the user’s email address on the page correctly. Things that are entirely trivial and obviously should Just Work take sufficient effort, and involve overcoming several failures, that just achieving the entirely ordinary seems like it deserves a celebration.

Clearly it is not normal for someone to celebrate the fact that as a result of typing stuff into the CSS file, the dotted border between the header of a table and the next row now displays correctly. So remote are programmers from the rich celebrations that life has to offer that we have to make our own.

There is a horrific irony to all this. Computers only do what they are programmed to do. But it is incredibly difficult to program them (correctly). What they are programmed to do is only an impression of what we intend them to do. In this case it is like trying to take an impression of a fossil with only wet tissue. So, on the one hand everything a computer does it has been programmed to do, on the other it is a monumentally difficult task to get the computer to do anything at all that is useful and correct.

Lovelace grasps this subtle point: “It can do whatever we know how to order it to perform”. It’s the “know how” that’s the tricky bit.

Go bug my cat!


Partly related to Francis Irving’s promise to avoid C, and partly related to the drinking at the recent PythonNW social, I wrote a version of /bin/cat in Go.

So far so good. It’s quite short.

The core loop has a curious feature:

for _, f := range args {
	func() {
		in, err := os.Open(f)
		if err != nil {
			fmt.Fprintf(os.Stderr, "%s\n", err)
		defer in.Close()
		io.Copy(os.Stdout, in)

The curious feature I refer to is that inside the loop I created an anonymous function and call it.

The earlier version of cat.go didn’t do that. And it was buggy. It looked like this:

for _, f := range args {
	in, err := os.Open(f)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
	defer in.Close()
	io.Copy(os.Stdout, in)

The problem with this is the defer. I’m creating one defer per iteration, and they don’t get run until the end of the function. So they just pile up until main returns. This is bad because each defer is going to close a file. If we try and cat 9999 copies of /dev/null we get this::

drj$ ./cat $(yes /dev/null | sed 9999q)
open /dev/null: too many open files

It fails because on Unix there is a limit to the number of simultaneous open files a process can have. It varies from a few dozen to a couple of thousand. When this version of cat opens too many files, it falls over.

In this case we failed because we ran out of a fairly limited resource, Unix file descriptors. But even without that, each defer allocates a small amount of memory (a closure, for example). So defer in loops requires (generally) a little anonymous function wrapper.

I tried rewriting the loop using a lambda and a tail call (see this git branch), but it doesn’t work. The defers still don’t run promptly. (and the tail call is awkward and I had to declare the loop variable on a separate line from the function itself because the scoping isn’t quite right)



Valve’s steam appears to be a package manager for installing Valve software (games). Part of steam on Linux is a shell script:

It turns out, if you’re not careful, if you try and uninstall steam or something… then this innocent 600 line shell script can kind of accidentally DELETE ALL YOUR USER FILES. Ha ha.

Much hilarity in the github issue.

At core the proximate issue is executing this command:

	rm -rf "$STEAMROOT/"*

The problem is that, perhaps in mysterious circumstances, STEAMROOT can be set to the empty string. Which means the command rm -fr "/"* gets executed. Which removes all the files that you have access to on the system (it might take its time doing this).

I’m working off this version of

First off, it’s 600 lines long. That, right there, should set the alarm bells ringing. No shell script should be that long. It’s just not a suitable language for writing more than a few lines in.

set -u, whilst a good idea in a lot of scripts, would not have helped here. As it happens, STEAMROOT is set, but set to the empty string.

"${STEAMROOT:?}", as suggested by one of the commentor’s in github, would have helped. The script will exit if STEAMROOT is unset or set to the empty string.

Immediately before this code there is a comment saying “Scary!”. So that’s another thing. If one programmer thinks the code is scary, then we should probably review the code. And make it less scary. Clearly adding an explicit check that STEAMROOT is set would have helped make it less scary.

It would also be a good idea to add a -- argument to rm to signify the end of the options. Otherwise if STEAMROOT starts with a «-» then it will trigger rm into thinking that it is an option instead of the directory to delete. So we should write:

    rm -fr -- "${STEAMROOT:?}"/*

STEAMROOT is assigned near the beginning of the file:

STEAMROOT="$(cd "${0%/*}" && echo $PWD)"

It is often problematic to use command substitution in an assignment. The problem being that the command inside the round brackets, cd "${0%/*}" && echo $PWD in this case, could fail. The shell script still carries on and assigns the stdout of the command to the variable. And if the command failed and produced no output then STEAMROOT will become the empty string.

Here would be a good place to explicitly check that STEAMROOT is not an empty string. : "${STEAMROOT:?}" will do, but if [ -z "$STEAMROOT" ] ; then exit 99; fi is more explicit.

set -e would have helped. If a command substitution is assigned to a variable and the command fails (exit code != 0) then the assignment statement fails and that will trigger set -e into exiting the script. It’s not ideal error checking, but it is better than nothing.

The code, as described by the comment above it, is trying to find out the location of the script. This is often problematic. There’s no portable way to find out. But as long as you’re in bash, and the script is explicitly is a bash script and uses various bashims, why not just use the relatively straightforward DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ) as recommended by this Stack Overflow answer. No need to pretend that $0 is set to anything useful. (all of the above still applies though)

The script is a bit enigmatic. Bits of it are written by someone who clearly knows shell scripting. The "{$0%/*"} thing to strip off the last component of a path is not common knowledge. But why not use dirname as the code later on in the script does? Correctly uses the portable equality operator single «=» in code like if [ "$STEAMEXE" = "steamcmd" ], but later on uses the bashism «==» and «[[». Clearly knows about the $( ... ) notation for command substitution, but then uses legacy (and yukhy) backquote syntax elsewhere. Carefully avoids using dirname (in the POSIX standard, and therefore very likely to be installed on any given Unix system), but then uses curl without checking (and curl isn’t installed on Ubuntu by default).

In summary: too long; attempting to locate directory containing script is problematic; doesn’t do enough checking (in particular, set -e).


Get every new post delivered to your Inbox.

Join 29 other followers