Nick Clegg on DRIP


I used WriteToThem to write to Nick Clegg, my MP; asking him how he intended to vote on the Data Retention and Investigatory Powers Bill. He didn’t say. But he did make this reply:

Thank you for your email regarding the Data Retention and Investigation Powers Bill.

The bill ensures that critical capabilities to fight crime and protect the public are maintained; it clarifies existing law without extending current powers. Interception and access to communications data are critical to the ability of our law enforcement and intelligence agencies to fight crime and protect the public. Furthermore, the bill makes clear that anyone providing a communications service to customers in the UK, regardless of where that service is provided from, should comply with lawful requests made under the Regulation of Investigatory Powers Act 2000. It also replaces the current Regulations under which domestic companies can be required to retain certain types of communications data for up to 12 months, so this may later be acquired by law enforcement and used in evidence. In addition, the bill will introduce further safeguards for the use of investigatory powers, building on our already stringent regime, to respond to criticisms raised by the European Court of Justice. If we fail to act immediately, there would be a critical gap in our investigative capabilities.

As a Liberal Democrat I believe that successive governments have neglected civil liberties as they claimed to pursue greater security. There is a real risk that we will suddenly be deprived of the legitimate means by which we keep people safe. Liberty and security must go hand in hand; we cannot enjoy our freedom if we are unable to keep ourselves safe. I therefore believe there is an urgent challenge facing us. As the Prime Minister has set out, we face the very stark prospect that powers we have taken for granted in the past, will no longer be available to us in the coming weeks.

Communications data and lawful intercept are now amongst the most useful tools available to us to prevent violence and bloodshed on Britain’s streets. Vital to the work of the police and the agencies is lawful intercept, which allows them to look, under warrant, at specific communications between individuals, who may be planning plots, and without retained communications data, we wouldn’t, for example, be able to piece together the web of relationships between members of the organised crime gangs responsible for trafficking drugs and vulnerable children and adults into our country.

It must be crucially noted that this bill has nothing to do with the so-called ‘Snooper’s Charter’. That was a Home Office proposal to store every website you’ve ever visited for a whole year. I blocked that last year and I’ve blocked every further attempt to bring it back. Moreover, this bill is about maintaining existing capabilities, not creating new powers. The legislation will restore the data retention powers we had before, but amend the law to do it in a more proportionate way, and it will help companies which currently provide assistance with UK intercept warrants to continue to do so.

Most importantly, I have only agreed to this emergency legislation because we can use it to kick-start a proper debate about freedom and security in the internet age. In the post-Snowdon age, people are, rightly demanding to know more about what the state does on our behalf. There are fundamental questions to be asked about the scope of existing powers; about whether the Regulation of Investigatory Powers Act has kept pace with technology; about how nation states grapple with a global internet; and about how we continue to protect security, privacy and liberties while safeguarding our security. We cannot answer these big questions on the hoof, and that is why I’ve insisted that the legislation only extends the existing powers for a temporary period. We’ve inserted a termination clause in the bill that means the legislation falls at the end of 2016, so the next government is forced to look again at these big issues.

Moreover, we will introduce new checks and balances to protect privacy and civil liberties for the future. We will establish, for the first time, a Privacy and Civil Liberties Oversight Board on the American model to ensure that Civil Liberties are properly considered in the formulation of government policy on counter-terrorism. We will radically cut the number of public bodies who have the right to approach phone and internet companies for your data. I don’t believe, for example, that local councils should unilaterally be able to access your communications data. From now on, councils will need to justify their requests first to a central body, and then a magistrate, and will not be able to approach phone and internet companies directly.

We will take practical steps to improve the accountability and transparency of our laws, by publishing regular transparency reports, listing new details about exactly how many warrants are issued, by whom, and for what purposes. The public will know more about how and why surveillance powers are administered on their behalf than ever before.

We have been working on this bill on a cross-party basis to achieve a balanced package which restores necessary powers while at the same time taking big new steps to defend our civil liberties.

I hope that this helps to answer your concerns. Please do not hesitate to get in touch again on this matter.

Yours sincerely,

Nick Clegg MP

Piping into shell may be harmful



curl https://thing | sh

It has become fashionable to see this kind of shell statement as a quick way of installing various bits of software (nvm, docker, salt).

This is a bad idea.

Imagine that due to bad luck the remote server crashes halfway through sending the text of the shell script, and a line that reads

rm -fr /usr/local/go

gets truncated and now reads:

rm -fr /

You’re hosed.

curl may write a partial file to the pipe and sh has no way of knowing.

Can we defend against this?

Initially I was pessimistic. How can we consider all possible truncations of a shell program? But then I realised, after a couple of false turns, that it’s possible to construct a shell script that is syntactically valid only when the entire script has been transmitted; truncating this special shell script at any point will result in a syntactically invalid script which shell will refuse to execute.

Moreover this useful property of being syntactically valid only when complete is not just a property of a small specially selected set of shell scripts, it’s entirely general. It turns out to be possible to transform any syntactically valid shell script into one that has the useful property:

any old script here

We can bracket the entire script with the grouping keyword { and }. Now if the script gets cutoff somewhere in the middle, the { at the beginning will be missing its matching } and be syntactically invalid. Shell won’t execute the partial script.

As long as the script in the middle is syntactically valid, then the bracketed script will be syntactically valid too.

Let’s call these curly brackets that protect the script from truncation, Jones truncation armour.

Clearly Jones truncation armour should be applied to all scripts that may be piped directly into shell.

Can it be applied as an aftermarket add-on? Yes it can!

{ echo { && curl https://thing && echo } ; } | sh

Maybe this is even better. It means that the consumer of the script doesn’t have to rely on the provider to add the Jones truncation armour. But it also doesn’t matter if the script is already armoured. It still works.

Coding is like Cooking


Coding is like cooking.

Well, not really. But a bit. This is not an article about how recipes are like programs, it’s about the role that cooking has in our personal lives and in society.

I can cook. A bit. Well enough that I can cook for my household, and friends that might drop by. I don’t always eat frozen pizza. Day to day cooking I can mostly do without a written recipe (spag bol, salmon and broccoli, that kind of thing), but when we entertain I’ll generally use a recipe; we own a few too many cookbooks and I can find recipes online. Perhaps one or two dishes I can make well enough that they’re actually good. So I can cook, but not well enough that anyone would pay me to do it. And as for being a chef there are probably other skills that professional cooks have that are part of the job that are simply not on my radar. Planning a menu, choosing suppliers, managing a kitchen.

I’m not suggesting that because I can cook a bit that I’ll be a cook. But conversely just because I can’t be a chef, that doesn’t mean that cooking is pointless. I don’t cook because it’s useful to the economy or because it helps me get a better job. I don’t do it as a hobby. I cook because it’s useful to me personally. It’s a sort of basic life skill.

I imagine most people are like me in this regard. They can cook, something. The amount of cooking that people do might vary. Some people will do it as a hobby, cooking things for their friends every week. Some people will do it professionally and cook for hundreds or thousands of people.

I would like programming to be like this.

I think most people can program. A bit. Not to a professional level, not to a standard where they would be comfortable getting paid to do it. Some people might like programming a bit more and do it as a hobby. Again, doesn’t mean they would be paid to do it.

You don’t have to be a programmer to program. Just like you don’t have to be a cook to cook.


I have friends who I think of as cooks. Some of them do it as a hobby, some professionally. One thing I’ve noticed is that they enjoy cooking, and they like to tinker. They’ll try cooking something just for fun on a Saturday morning. The professionals will try some new technique or ingredient or idea because it will expand their power. I don’t do that. Not with cooking anyway. But I do with programming. I’ll write a browser-based Logo implementation to learn a bit about SVG and JavaScript. Or I’ll learn a new language because it seems fun and might stretch me intellectually.

Are there problems with this analogy? Yes there are:

  • Cooking is useful in itself. The result is usually a meal; you can eat it. This is not so clearly true of programming. There are useful programs to be written, but by and large the kinds of programs that non-professional coders write are not all that useful (yet; see below).
  • Terminology. Lots of people (professionals and hobbyists alike) seem to think that when you say “teach people to program” you mean “train people to be professional programmers”. We don’t. Or at least, I don’t. No more than “teaching people to cook” means “teaching people to be cooks”.
  • Access

    The access route to cooking is pretty straightforward. Most homes and (all?) schools have a hob or a cooker or a microwave. You can cook something with that. I don’t have to buy a stainless steel counter and professional range to cook. You shouldn’t need “pro-level” tools to program. You shouldn’t have to buy a specialist computer and learn how to use an editor like vi, emacs, or TextMate.

    So I think we have to make the access route to programming simpler.

    Tools like ScraperWiki’s Code in your Browser tool help (full disclosure: I work for the company that made this), as may things like CoffeePlay (note I said things like CoffeePlay, which is a whimsical tool created over a weekend, not a serious product). You can start programming using just your browser. There is a browser based version of MIT’s Scratch programming environment. The Raspberry Pi helps.

    At the moment, I think these things show possible futures. They are hints at how we can make coding more accessible. But I think it is possible to make the route to programming simpler. A child can take the first steps in cooking by mixing flour and butter, and putting scones in the oven. We need to make programming just as easy and accessible.

    We need to make it easier to do more things with code. What I mean here is more hackable things in the spirit of the maker community, hack spaces, and so on. It’s kind of neat that I can log into my Kobo ereader and modify the software, it’s a shame I can’t do that with my Kindle so easily. I really love my label printer, but I wish I could hack the firmware on it.

    I don’t think everyone should be a cook, but I think everyone should cook.

    Traditional approach to ATtiny programming


    I recently bought an Adafruit Gemma. It’s a little programming board that is slightly bigger than a 10p coin and it costs about GBP 7.

    It uses an ATtiny85 micro and is Arduino compatible, so the way you’re encouraged to program is to use the Arduino GUI tools and all that good stuff.

    By “traditional approach” I mean grumpy old man approach. I don’t like GUIs much. I can’t use vi and the rendering of the font in the editor is terrible. And the syntax highlighting burns my eyes.

    So you can just use the command line tools, right? Right. On Ubuntu you can apt-get install gcc-avr and then use the avr-gcc compiler to compile your C code. You’ll need avr-objcopy (from the binutils-avr package) to convert your .elf file into an Intel .hex file, and you’ll need avrdude (from the avrdude package) to flash the device. The gory details are captured in this Makefile, and I got those gory details by switching the Arduino GUI into its verbose mode and watching it compile my project.

    My first demo project is also an exercise in avoiding the Arduino libraries. Mostly because when I was working out how to use the command line tools, I didn’t want the hassle of dealing with multiple files and libraries and things.

    So this is also an example of how to program the ATtiny85 (and more or less any AVR type micro) without using heavyweight libraries. The Gemma has a built-in red LED on PB1. This was definitely one of the things that attracted me to the Gemma. I can program it do something without needing to plug any extra hardware in. Specifically, I can flash the LED.

    Flashing an LED is a matter of using a GPIO pin and driving it high (on) and low (off). The assembler programmer would do that with the SBI (Set BIt) and CBI (Clear BIt) instructions. So I’m thinking “Can we have reasonable looking C code generate The Right Instruction?”.

    The C code to set a bit is generally of the form *p |= b where p is a pointer to some memory location and b is a number with a single bit set (a power of 2 in other words). Similarly the C code to clear a bit is *p &= ~b. As long as we give the compiler enough information, it should be able to compile the code *p |= b into an SBI instruction.

    And so it can. Through some fairly tedious but also fairly ordinary C macros, I can write BIT_SET(PORTB, 1) to set pin 1 or PORTB (PB1, the pin with the LED attached), and it gets converted into roughly: *(volatile uint8_t *)0x38 |= 2; which is basically saying modify memory location 0x38 by setting its bit 1. In a little oddity of the AVR architecture the SBI and CBI instructions operate on IO addresses which are at memory locations 0x20 onwards. The upshot of this is that memory location 0x38 is modified with a SBI 0x18 instruction (this mystified me for about 2 hours last night, and I realised what was wrong just as I was drifting off to sleep).

    Because in the code *(volatile uint8_t *)0x38 |= 2; both the location, 0x38, and the value, 2, are constant, the compiler has everything it needs to generate the right SBI instruction. And it does!

    drj$ avr-objdump -d main.elf 2>&1 | sed -n '/<main>:/,$p' | sed 9q
    00000040 <main>:
      40:	1f 93       	push	r17
      42:	b9 9a       	sbi	0x17, 1	; 23
      44:	c1 9a       	sbi	0x18, 1	; 24
      46:	88 ec       	ldi	r24, 0xC8	; 200
      48:	90 e0       	ldi	r25, 0x00	; 0
      4a:	f2 df       	rcall	.-28     	; 0x30 <delay>
      4c:	c1 98       	cbi	0x18, 1	; 24
      4e:	88 ec       	ldi	r24, 0xC8	; 200

    You can see at the beginning of the disassembly of main the SBI 0x17, 1 instruction which is as a result of the macro BIT_SET(DDRB, 1) (setting the pin to be a digital output). And you can see SBI 0x18, 1 to drive the pin high and light the LED and CBI 0x18, 1 to drive the pin low. The compiler has even subtracted 0x20 from the addresses.

    avr-gcc -Os FTW!

    classy enumerations


    An enumeration is a term that usually refers to a type consisting of a finite number of distinct members. The members themselves can be tested for equality, but usually their particular value is not important.

    Maybe you’re modelling a Sphex wasp and you have a state variable with values NOTNEARHOME, JUSTLEFTHOME. You could represent that with an enumeration.

    In C the enum keyword assigns (usually small) integers to constant identifiers. It is problematic, chiefly because the members of the enumeration are really just integers. After enum { Foo; }, then code like Foo / 5 is valid (note: valid, not sensible).

    In Python you could do essentially the same thing:

    if self.state == NOTNEARHOME:
        if 'spider' in self.inventory:
            # head towards home
            # look for juicy spiders

    You do see this style (ast.PyCF_ONLY_AST Note 1), but it has the same problems as enum in C. The values are just integers (so, for example, print self.state will print 0, or 1).

    You could use strings (like decimal.ROUND_HALF_EVEN):

    NOTNEARHOME = 'notnearhome'
    # and so on...

    That’s better because now I might have a clue if a print out a value and it’s 'notnearhome', but it’s only a little bit better, because you still might accidentally use the value innappropriately (opt = decimal.ROUND_HALF_EVEN; opt += 'foo').

    I have a proposal:

    Use a class!

    class NOTNEARHOME: pass         # Note 2
    class JUSTLEFTHOME: pass

    Let’s call this classy enumerations.

    Classy enumerations have the advantage that we don’t need to manually assign numbers or strings. Values like Mood.PUZZLED and Mood.CONFUSED (which are actually classes) will be unique, so can be tested using == or is correctly.

    With classy enumerations we get an error if we accidentally use them in an expression:

    >>> PUZZLED+1
    Traceback (most recent call last):
      File "", line 1, in 
    TypeError: unsupported operand type(s) for +: 'classobj' and 'int'

    And to wrap up:

    class True: pass
    class False: pass

    This article was inspired by looking at some of Peter Waller‘s code who seems to have invented the idea of using classes for enumerations.

    Note 1

    Yes this value matches a value in the C header file. Maybe that has some merit, but it doesn’t make for a very Pythonic interface.

    Note 2

    The body of a class definition is just a block of code. When that body is just a simple statement, it can share the line with the class declaration. Hence, class NOTNEARHOME: pass is a compact complete class definition. If you’re in a mood for documentation, replace “pass” with a docstring.

    Teach everyone to program


    The microcomputer was invented only a generation ago, it is now in hundreds of devices and gadgets in every home.

    We are on the brink of a revolution. A revolution as significant as the printing press. It was hundreds of years between the press and universal literacy (still not quite achieved in the UK, our adult literacy rate is 99%), but we now live in a society where so much information is written and so much commerce and social interaction takes place in writing that you are at a severe disadvantage if you cannot read and write. We are not all writers, but we can all write. Poets can move entire Nations with carefully crafted written words, but even if most of us can’t achieve that, we can at least write a note to our milkman asking for 2 more pints on Saturday (Note 1).

    Imagine if Gove (Note 2) suggests that we only teach the gifted to write. Only the playwrights, the speechwriters, the journalists, and the poets will write. After all, all the good stuff is written by them anyway. I hope you can see that this would be madness.

    I feel the same way about code and programming. There is a poetry to code; the poets of programming write code with concision and precision: the structure writ clear on the fan-folded page. There is already a rich literature of programming. github overflows with the pulp fiction of the professional and amateur hack alike; likely we will find Lovecraftian horrors there too, lurking, ready to turn our minds into a pretzel. But at the moment this culture is the culture of an elite class. Everyone should be able to program. Most people will not be poets. Most people will not be programmers, but that should not stop us from teaching them to program.

    In the future code will be woven into the fabric of our society, just as the written word is woven now. We don’t teach people to read and write because it will be helpful to them in their future career. We teach them because it is inconceivable that they can function without basic literacy. It is inconceivable that in the future we will be able to function without basic coding.

    This is why I’m excited about teaching kids to program. Lots of grassroots initiatives to teach programming, like Software Carpentry, Raspberry Pi, Young Rewired State. Let’s build the future now. Teach everyone to program.

    Note 1

    I live in a quaint postcode where I can still get milk delivered to my doorstep. And I do so.

    Note 2

    Hello readers from the future! Gove was responsible for education policy in the United Kingdom (Note 3) for a brief period in the early 21st Century.

    Note 3

    Hello readers from the more distant future. The United Kingdom was a nation consisting of various bits of islands in an archipelago off the North West coast of Europe.

    Women of #movember


    In the interests of brevity the title says “women” but I mean to include all partners of people growing a mo’ this month.

    I am growing a moustache to support a local prostate cancer group.

    The reality is that growing a moustache is not hard. I just have to not shave a certain part of my face for a while. Most of the hardship is endured by my partner Philippa who has to put up with bristly kisses for a month.

    Support is vital for success. Support for survivors of cancer, and support for sufferers of cancer. Support for people raising money.

    I’d like to thank all the partners out there that are supporting their men growing a moustache. I would not be able to do this without the support of my family and friends, and I’m sure others feel similarly.

    You can give your support here:

    Explaining p += q in Python


    If you’re a Python programmer you should know about the augmented assignment statements:

    i += 1

    This adds one to i. There is a whole host of these augmented operators (-=, *=, /=, %= etc).

    Aside: I used to call these assignment operators which is the C terminology, but in Python assignment is a statement, not an expression (yay!): you can’t go while i -= 1 (and this is a Good Thing).

    An augmented assignment like i += 1 is often described as being the same as i = i + 1, except that i can be a complicated expression and it will not be evaluated twice.

    As Julian Todd pointed out to me (a couple of years ago now), this is not quite right when it comes to lists.

    Recall that when p and q are lists, p + q is a fresh list that is neither p nor q:

    >>> p = [1]
    >>> q = [2]
    >>> r = p + q
    >>> r
    [1, 2]
    >>> r is p
    >>> r is q

    So p += q should be the same as p = p + q, which creates a fresh list and assigns a reference to it to p, right?

    No. It’s a little bit tricky to convince yourself of this fact; you have to keep a second reference to the original p (called op below):

    >>> p = [1]
    >>> op = p
    >>> p += [2]
    >>> p
    [1, 2]
    >>> op
    [1, 2]
    >>> p is op

    Here it is in pictures:

    Because of this, it’s slightly harder to explain how the += assignment statement behaves. For numbers we can explain it by breaking it down into a + operator and an assignment, but for lists this explanation fails because it doesn’t explain how p (in our p += q example) retains its identity (the curious will have already found out that+= is implemented by calling the __iadd__ method of p).

    What about tuples?

    When p and q are tuples the behaviour of += is more like numbers than lists. A fresh tuple is created. It has to be, since you can’t mutate a tuple.

    This kind of issue, the difference between creating a fresh object and mutating an existing one, lies at the heart of understanding the P languages (perl, Python, PHP, Ruby, JavaScript).

    The keen may wish to fill in this table:

    p q p + q p += q
    list list fresh p mutated
    tuple tuple fresh fresh
    list tuple ? ?
    tuple list ? ?



    One of the benefits of getting the train from Sheffield to Liverpool is the beautiful views of the Peak District that I don’t pay enough attention to.

    Winter is coming, consequently, it’s just the right time to see the sunrise:

    copper bronze dashed across the sky
    blots and streaks of morning’s blue
    a speck of an aeroplane drifted lazily
    the buildings of flats and industry were silhouetted
    brick chimneys thrust industrially upward
    we arrived in Manchester.

    On compiling 34 year old C code


    The 34 year old C code is the C code for ed and sed from Unix Version 7. I’ve been getting it to compile on a modern POSIXish system.

    Some changes had to be made. But not very many.

    The union hack

    sed uses a union to save space in a struct. Specifically, the reptr union has two sub structs that differ only in that one of them has a char *re1 field where the other has a union reptr *lb1. In the old days it was possible to access members of structs inside unions without having to name the intermediate struct. For example the code in the sed implementation uses rep->ad1 instead of rep->reptr1.ad1. That’s no longer possible (I’m pretty sure this shortcut was already out of fashion by the time K&R was published in 1978, but I don’t have a copy to hand).

    I first changed the union to a struct that had a union inside it only for the two members that differed:

    struct	reptr {
    		char	*ad1;
    		char	*ad2;
    	union {
    		char	*re1;
    		struct reptr	*lb1;
            } u;
    		char	*rhs;
    		FILE	*fcode;
    		char	command;
    		char	gfl;
    		char	pfl;
    		char	inar;
    		char	negfl;
    } ptrspace[PTRSIZE], *rep;

    The meant changing a few “union reptr” to “struct reptr”, but most of the member accesses would be unchanged. ->re1 had to be changed to ->u.re1, but that’s a simple search and replace.

    It wasn’t until I was explaining this ghastly use of union to Paul a day later that I realised the union is a completely unnecessary space-saving micro-optimisation. We can just have a plain struct where only one of the two fields re1 and lb1 were ever used. That’s much nicer, and so is the code.

    The rise of headers

    In K&R C if the function you were calling returned int then you didn’t need to declare it before calling it. Many functions that in modern times return void, used to return int (which is to say, they didn’t declare what they returned, so it defaulted to int, and if the function used plain return; then that was okay as long as the caller didn’t use the return value). exit() is such a function. sed calls it without declaring it first, and that generates a warning:

    sed0.c:48:3: warning: incompatible implicit declaration of built-in function ‘exit’ [enabled by default]

    I could declare exit() explicitly, but it seemed simpler to just add #include <stdlib.h>. And it is.

    ed declares some of the library functions it uses explicitly. Like malloc():

    char	*malloc();

    That’s a problem because the declaration of malloc() has changed. Now it’s void *malloc(size_t). This is a fatal violation of the C standard that the compiler is not obliged to warn me about, but thankfully it does.

    The modern fix is again to add header files. Amusingly, version 7 Unix didn’t even have a header file that declared malloc().

    When it comes to mktemp() (which is also declared explicitly rather than via a header file), ed has a problem:

    tfname = mktemp("/tmp/eXXXXX");

    2 problems in fact. One is that modern mktemp() expects its argument to have 6 “X”s, not 5. But the more serious problem is that the storage pointed to by the argument will be modified, and the string literal that is passed is not guaranteed to be placed in a modifiable data region. I’m surprised this worked in Version 7 Unix. These days not only is it not guaranteed to work, it doesn’t actually work. SEGFAULT because mktemp() tries to write into a read-only page.

    And the 3rd problem is of course that mktemp() is a problematic interface so the better mkstemp() interface made it into the POSIX standard and mktemp() did not.

    Which brings me to…

    Unfashionable interfaces

    ed uses gtty() and stty() to get and set the terminal flags (to switch off ECHO while the encryption key is read from the terminal). Not only is gtty() unfashionable in modern Unixes (I replaced it with tcgettattr()), it was unfashionable in Version 7 Unix. It’s not documented in the man pages; instead, tty(4) documents the use of the TIOCGETP command to ioctl().

    ed is already using a legacy interface in 1979.


    Get every new post delivered to your Inbox.

    Join 29 other followers