Archive for March, 2014

Piping into shell may be harmful



curl https://thing | sh

It has become fashionable to see this kind of shell statement as a quick way of installing various bits of software (nvm, docker, salt).

This is a bad idea.

Imagine that due to bad luck the remote server crashes halfway through sending the text of the shell script, and a line that reads

rm -fr /usr/local/go

gets truncated and now reads:

rm -fr /

You’re hosed.

curl may write a partial file to the pipe and sh has no way of knowing.

Can we defend against this?

Initially I was pessimistic. How can we consider all possible truncations of a shell program? But then I realised, after a couple of false turns, that it’s possible to construct a shell script that is syntactically valid only when the entire script has been transmitted; truncating this special shell script at any point will result in a syntactically invalid script which shell will refuse to execute.

Moreover this useful property of being syntactically valid only when complete is not just a property of a small specially selected set of shell scripts, it’s entirely general. It turns out to be possible to transform any syntactically valid shell script into one that has the useful property:

any old script here

We can bracket the entire script with the grouping keyword { and }. Now if the script gets cutoff somewhere in the middle, the { at the beginning will be missing its matching } and be syntactically invalid. Shell won’t execute the partial script.

As long as the script in the middle is syntactically valid, then the bracketed script will be syntactically valid too.

Let’s call these curly brackets that protect the script from truncation, Jones truncation armour.

Clearly Jones truncation armour should be applied to all scripts that may be piped directly into shell.

Can it be applied as an aftermarket add-on? Yes it can!

{ echo { && curl https://thing && echo } ; } | sh

Maybe this is even better. It means that the consumer of the script doesn’t have to rely on the provider to add the Jones truncation armour. But it also doesn’t matter if the script is already armoured. It still works.

Coding is like Cooking


Coding is like cooking.

Well, not really. But a bit. This is not an article about how recipes are like programs, it’s about the role that cooking has in our personal lives and in society.

I can cook. A bit. Well enough that I can cook for my household, and friends that might drop by. I don’t always eat frozen pizza. Day to day cooking I can mostly do without a written recipe (spag bol, salmon and broccoli, that kind of thing), but when we entertain I’ll generally use a recipe; we own a few too many cookbooks and I can find recipes online. Perhaps one or two dishes I can make well enough that they’re actually good. So I can cook, but not well enough that anyone would pay me to do it. And as for being a chef there are probably other skills that professional cooks have that are part of the job that are simply not on my radar. Planning a menu, choosing suppliers, managing a kitchen.

I’m not suggesting that because I can cook a bit that I’ll be a cook. But conversely just because I can’t be a chef, that doesn’t mean that cooking is pointless. I don’t cook because it’s useful to the economy or because it helps me get a better job. I don’t do it as a hobby. I cook because it’s useful to me personally. It’s a sort of basic life skill.

I imagine most people are like me in this regard. They can cook, something. The amount of cooking that people do might vary. Some people will do it as a hobby, cooking things for their friends every week. Some people will do it professionally and cook for hundreds or thousands of people.

I would like programming to be like this.

I think most people can program. A bit. Not to a professional level, not to a standard where they would be comfortable getting paid to do it. Some people might like programming a bit more and do it as a hobby. Again, doesn’t mean they would be paid to do it.

You don’t have to be a programmer to program. Just like you don’t have to be a cook to cook.


I have friends who I think of as cooks. Some of them do it as a hobby, some professionally. One thing I’ve noticed is that they enjoy cooking, and they like to tinker. They’ll try cooking something just for fun on a Saturday morning. The professionals will try some new technique or ingredient or idea because it will expand their power. I don’t do that. Not with cooking anyway. But I do with programming. I’ll write a browser-based Logo implementation to learn a bit about SVG and JavaScript. Or I’ll learn a new language because it seems fun and might stretch me intellectually.

Are there problems with this analogy? Yes there are:

  • Cooking is useful in itself. The result is usually a meal; you can eat it. This is not so clearly true of programming. There are useful programs to be written, but by and large the kinds of programs that non-professional coders write are not all that useful (yet; see below).
  • Terminology. Lots of people (professionals and hobbyists alike) seem to think that when you say “teach people to program” you mean “train people to be professional programmers”. We don’t. Or at least, I don’t. No more than “teaching people to cook” means “teaching people to be cooks”.
  • Access

    The access route to cooking is pretty straightforward. Most homes and (all?) schools have a hob or a cooker or a microwave. You can cook something with that. I don’t have to buy a stainless steel counter and professional range to cook. You shouldn’t need “pro-level” tools to program. You shouldn’t have to buy a specialist computer and learn how to use an editor like vi, emacs, or TextMate.

    So I think we have to make the access route to programming simpler.

    Tools like ScraperWiki’s Code in your Browser tool help (full disclosure: I work for the company that made this), as may things like CoffeePlay (note I said things like CoffeePlay, which is a whimsical tool created over a weekend, not a serious product). You can start programming using just your browser. There is a browser based version of MIT’s Scratch programming environment. The Raspberry Pi helps.

    At the moment, I think these things show possible futures. They are hints at how we can make coding more accessible. But I think it is possible to make the route to programming simpler. A child can take the first steps in cooking by mixing flour and butter, and putting scones in the oven. We need to make programming just as easy and accessible.

    We need to make it easier to do more things with code. What I mean here is more hackable things in the spirit of the maker community, hack spaces, and so on. It’s kind of neat that I can log into my Kobo ereader and modify the software, it’s a shame I can’t do that with my Kindle so easily. I really love my label printer, but I wish I could hack the firmware on it.

    I don’t think everyone should be a cook, but I think everyone should cook.

    Traditional approach to ATtiny programming


    I recently bought an Adafruit Gemma. It’s a little programming board that is slightly bigger than a 10p coin and it costs about GBP 7.

    It uses an ATtiny85 micro and is Arduino compatible, so the way you’re encouraged to program is to use the Arduino GUI tools and all that good stuff.

    By “traditional approach” I mean grumpy old man approach. I don’t like GUIs much. I can’t use vi and the rendering of the font in the editor is terrible. And the syntax highlighting burns my eyes.

    So you can just use the command line tools, right? Right. On Ubuntu you can apt-get install gcc-avr and then use the avr-gcc compiler to compile your C code. You’ll need avr-objcopy (from the binutils-avr package) to convert your .elf file into an Intel .hex file, and you’ll need avrdude (from the avrdude package) to flash the device. The gory details are captured in this Makefile, and I got those gory details by switching the Arduino GUI into its verbose mode and watching it compile my project.

    My first demo project is also an exercise in avoiding the Arduino libraries. Mostly because when I was working out how to use the command line tools, I didn’t want the hassle of dealing with multiple files and libraries and things.

    So this is also an example of how to program the ATtiny85 (and more or less any AVR type micro) without using heavyweight libraries. The Gemma has a built-in red LED on PB1. This was definitely one of the things that attracted me to the Gemma. I can program it do something without needing to plug any extra hardware in. Specifically, I can flash the LED.

    Flashing an LED is a matter of using a GPIO pin and driving it high (on) and low (off). The assembler programmer would do that with the SBI (Set BIt) and CBI (Clear BIt) instructions. So I’m thinking “Can we have reasonable looking C code generate The Right Instruction?”.

    The C code to set a bit is generally of the form *p |= b where p is a pointer to some memory location and b is a number with a single bit set (a power of 2 in other words). Similarly the C code to clear a bit is *p &= ~b. As long as we give the compiler enough information, it should be able to compile the code *p |= b into an SBI instruction.

    And so it can. Through some fairly tedious but also fairly ordinary C macros, I can write BIT_SET(PORTB, 1) to set pin 1 or PORTB (PB1, the pin with the LED attached), and it gets converted into roughly: *(volatile uint8_t *)0x38 |= 2; which is basically saying modify memory location 0x38 by setting its bit 1. In a little oddity of the AVR architecture the SBI and CBI instructions operate on IO addresses which are at memory locations 0x20 onwards. The upshot of this is that memory location 0x38 is modified with a SBI 0x18 instruction (this mystified me for about 2 hours last night, and I realised what was wrong just as I was drifting off to sleep).

    Because in the code *(volatile uint8_t *)0x38 |= 2; both the location, 0x38, and the value, 2, are constant, the compiler has everything it needs to generate the right SBI instruction. And it does!

    drj$ avr-objdump -d main.elf 2>&1 | sed -n '/<main>:/,$p' | sed 9q
    00000040 <main>:
      40:	1f 93       	push	r17
      42:	b9 9a       	sbi	0x17, 1	; 23
      44:	c1 9a       	sbi	0x18, 1	; 24
      46:	88 ec       	ldi	r24, 0xC8	; 200
      48:	90 e0       	ldi	r25, 0x00	; 0
      4a:	f2 df       	rcall	.-28     	; 0x30 <delay>
      4c:	c1 98       	cbi	0x18, 1	; 24
      4e:	88 ec       	ldi	r24, 0xC8	; 200

    You can see at the beginning of the disassembly of main the SBI 0x17, 1 instruction which is as a result of the macro BIT_SET(DDRB, 1) (setting the pin to be a digital output). And you can see SBI 0x18, 1 to drive the pin high and light the LED and CBI 0x18, 1 to drive the pin low. The compiler has even subtracted 0x20 from the addresses.

    avr-gcc -Os FTW!