Archive for May, 2009

Anonymous things in Python


This post of pydanny prompted me to flesh this article out (previously it had just been floating around in my mind). pydanny’s article reminded me of when I first learned about lambda and how scary it was. That was in about 1988 when I was trying to get to grips with XLISP on the Atari ST (hah! a Lisp dialect so old it still spells its name with capital letters. sweet). Previous to XLISP I had been programming in (Sinclair) BASIC, z80 machine code, and 68000 assembler. Oh, and FORTH. Maybe you can imagine how scary and alien a concept like lambda was. I can just barely remember how confusing it was. I don’t think the confusion really passed until I started dabbling with ML, in 1991. By the time I started dabbling with the lambda calculus in 1993 I was actually pretty comfortable with it. So, only 5 years of exposure to get lambda to get comfy.

Of course now, I know that:

A lambda is a simply function with no name.

I emphasise the simply because I feel that that’s where the problem of understanding lay (for me). In BASIC and FORTH it was impossible to have a function without a name. In fact, the very thought was literally inconceivable. At least for a few years. Even in machine code where functions don’t have names they do have concrete locations, and they have names in the listing.

It wasn’t until many years later, in fact perhaps relatively recently, that I realised that there was a sort of hierarchy of things that must be named, and things that need not be named. For example, an expression, like «a**2 + b**2 + c**2» is an anonymous value, a value that does not need a name. More importantly, so are all the sub-expressions inside that expression. What would a language look like if it didn’t have anonymous values? Well, something like this:

a2 = a**2
b2 = b**2
c2 = c**2
s = a2+b2
s = s + c2

Do you see? Each expression must be given a name (by assigning it to a named variable), and no expression can contain a sub-expression, because that sub-expression would be anonymous.

Programming in assembler is a bit like this. The names are the registers and every expression has to go in a register. If you run out, you have to rename everything by spilling to the stack. It’s partly why it can be so tedious.

So almost everything worthy of being called a language has anonymous values. At least simple values, like numbers. In the BASIC era it was common for strings to be named only. The kinds of restrictions you had to deal with were things like not being able to create a string from part of an existing string. You had to first create a named variable that was the target string, then copy part of the source string into the target string. Maybe you had some string expressions but there were restrictions on where you could use them. For example maybe you could only pass named strings to functions, so you could go «T$=MID$(A$,12,16):PROCFOO(T$)» but not pass the MID$ expression directly: «PROCFOO(MID$(A$,12,16))» (bad bad bad). Life was hard, and we licked coal for breakfast.

Arrays were another thing where you had to name them and couldn’t pass them to functions (for example). AWK is still like this (arrays cannot be returned from a function, for example).

Speaking of arrays, an element of an array is a bit like an anonymous variable:

a[i] = x

Above, a[i] specifies the place where the computed value, x is this case, goes. I say “a bit like” because you could equally well think of the element being named by the array name and an small integer as a pair. A place where we can store a value that changes is called a variable. You knew that right? It’s just that most people think of named variables when you say variable. The generalised term for “anything you can put on the left of an assignment” is lvalue. A term you mostly see in the C programming language but often elsewhere too. Common Lisp calls such things, places. How sensible.

So the hierarchy of things you need not name starts something like:

  • simple numeric expressions
  • strings
  • arrays (including hash-tables, and so on)
  • objects (incuding structs, records, pairs, and so on)
  • functions
  • places (actually, not sure where this should go really)
  • As you go down the list, you see fewer and fewer languages that give you anonymous versions of the thing in question. Although these days we probably all agree that any decent language will have anonymous versions of all of the above. Python does, and that’s a Good Thing.

    One reason for arranging things this way is that we can use it to both explain and motivate things like lambdas. A lambda is an anonymous function, and you remember how useful it was when you realised you could have string expressions and you didn’t have to store intermediate strings in variables? You could just manipulate strings without having to name them? Well, the same is true of functions.

    So what else could and should be anonymous? Well we can look to other languages for inspiration. ML has anonymous types. Java has anonymous classes. So perhaps the list can continue:

  • classes
  • types
  • (note that a class is a sort of type, so this order has a natural feel to it).

    So everything I said about anonymous functions applies to anonymous classes and anonymous types. They’re great, and every language needs them. Basically my philosophy here is that if there’s a thing in a language and you can name it, then you should also be able to have anonymous versions of those things. That will make the language better.

    So what other sorts of things does Python have that we haven’t removed the names from yet? Modules. For a long while I wished that Python had anonymous modules, a way of getting hold of a module without it polluting your namespace. Mostly I just wished for this because it would make Python more orthogonal, or better as I like to say. Of course modules in Python are first class citizens. Once you’ve imported the module struct, then struct refers to the module as a first class value, and it can be passed around and so on. That’s how help(struct) works. Recently I found both the way to get at modules anonymously (without importing them into a namespace) , and a reason why you might want to.

    I’m a tidy sort of guy so when I program in Python I put my imports only in the functions that need them (a habit I picked up from Command Line Warriors but buggered if I can find the original inspiration). Well, sometimes I am.

    I often find myself debugging my Python by the time honoured tradition of inserting print statements. Because of my tidy habit, I quite often find that the sys module is not in scope (in fact, often the only function that has «import sys» is main. So this fails:

    print >>sys.stderr, "length:", l, "data:" x

    And I’ll be damned if I’m going to add an import sys and forget to remove it later. But this works:

    print >>__import__('sys').stderr, "length:", l, "data:" x

    Not a very nice syntax though, is it?

    Extreme Pedantry For The Win!


    Last month I got a Penalty Charge Notice from Islington. I am the registered keeper of the vehicle in this alleged contravention:

    A car, going the wrong side of a "keep left" sign.

    I think normally I would’ve just sucked it up and paid the fine, but… they caught me in the wrong mood. Clearly the street furniture is arranged so as to maximise ticketing revenue from tourists. This appears immediately after a left-turn to enter the road. In the first picture (left) the brakes are on; the car is stopped because the driver is wondering how to interpret this fantastic and unusual display of road furniture. Locals will know about already and just drive around.

    I wrote the following letter:

    I have received a Penalty Charge Notice from Islington.

    “LLATLA 2003” is a reference to primary legislation London Local Authorities and Transport for London Act 2003.

    On reviewing the relevant legislation I find that Islington fail to meet their statutory obligations with regard to the penalty charge notice (LLATLA 2003 Section 4 subection (8)).  I find that:
    – there is failure to state the grounds on which Islington believe that the penalty charge is payable.  In particular, Islington should state whether their belief is that Section 4 subsection (5)(a) applies, or Section 4 subsection (5)(b) applies (referring to LLATLA 2003).  I am content that Section 4 subsection (7) does not apply;
    – there is failure to state that the person on whom the notice is served may be entitled to make representations under paragraph 1 of Schedule 1 of LLATLA 2003;
    – there is failure to specify the form in which any such representations are to be made.

    If Islington intends to pursue this matter further, please send a revised penalty charge notice.

    Today I got a letter from Steven Prieditis, Correspondence and Appeals Office, Islington Council. There’s some blurb, and then:

    Due to an error when issuing the PCN thsi ticket has been cancelled.



    I’m actually slightly disappointed. In the (now cancelled) PCN they allege: “Failing to drive in the direction shown by the arrow on a blue sign”.

    That contravention did not take place. Referring to diagram 606 of The Traffic Signs Regulations and General Directions 2002, we can plainly see that the alleged offence would be failing to observe a sign like this:


    “vehicular traffic must proceed in the direction indicated by the arrow”.

    You can just about make out the blue sign in the pictures at the top of the article. It’s this:


    (Diagram 610 of the regulations). And the description of that sign is “vehicular traffic must comply with the requirements prescribed in regulation 15” (of The Traffic Signs Regulations and General Directions 2002).

    Wrong sign. Didn’t happen.

    PS I hereby license you to do whatever you like with the letter I wrote.

    Earth, again


    So unless you’re using OS X (or, apparently, Firefox 3.5b4 on Vista), my earlier article and MDIS and embedded colour profiles and what-not must’ve seemed a little bit mysterious. The two pictures of Earth will have looked the same (on computers that ignore the ICC profile embedded in the PNG).

    It turns out that Apple’s ColorSync Utility will let you change an image’s ICC profile: either editing the image to fit the new profile, or just slapping a different profile on (I discovered this by reading the Help, making it the second useful fact I’ve learnt from OS X Help). Strangely, ColorSync can also resize an image. And it’s a magnification tool. Anyway, remapping the images into sRGB allows me to show you a simulated version of what I see. Your computer will ignore the embedded sRGB profile, but that’ll be okay, because it’ll look roughly right. That’s what sRGB is designed to do.




    The top image is a simulation of the raw MDIS camera values in a linear space. The bottom image is the corrected version using the ICC profile I showed in the earlier article.

    The middle image is the “just blat the pixels onto the graphics card” version that ignores all gamma and ICC profiles. As you can see, in terms of black level, it’s hard to distinguish its background from your display’s true black unless you put a true black right next to it.

    PS This is supposed to be Earth. Does anyone recognise what we’re looking at? Technically there’s enough metadata in the original IMG files for me to answer that question, since the exact time of the exposure is recorded as well as exactly where the camera was pointing, but like that’s going to happen.

    My laptop bag


    Just got back from a weekend on the farm (actually that was a few weekends ago, but I only just got round to cleaning up the article). Thought it might be fun to clear out my laptop bag. My laptop bag is my man-bag, it’s a portable version of my life. The bag is a fairly ordinary looking black affair made out of the hide of baby seals. I blagged it from my mum after she retired from being a power manager. It’s clearly very well made as it suffers much abuse and has lasted very well indeed. Want to look inside?

    Short IKEA pencil with big wedge-shaped rubber on the back. Ideal for sudoku (not that I ever do them).

    A borrowed laser pointer. Doesn’t work.

    10 Ultra Soft white tissues from FSC Mixed Sources.

    Half a packet of Wrigley’s Extra Peppermint Sugarfree Gum. Odd, since I don’t chew gum. Also, I don’t particular like sugar-free products; it’s just another term for “contains aspartame”.

    A badge (not a conference badge, for once).

    A Keyboard Anthology, Edited by Howard Ferguson, Book 1 (Grades 1 & 2), ASBRM. You know, for the pianoforte.

    The handmade slip case for my laptop.

    Pilot V5. The only writing implement I can depend on.

    Pair of cheap Aiwa earphones.

    A cable with a headphone jack at both ends. This cable languished in my boxes of crud for years until I unexpectedly found a good use for it. More on that in a later article perhaps.

    Darwin, “Voyage of the Beagle”. One of many half-read books, but this one is the one I currently carry around.

    Dog eared fag end of a pack of almost uselessly small Post-It notes. Each one covers only 6 keys on my keyboard.

    A blue USB memory stick that takes an SD card. A 2GB SD card, in fact.

    Some keys.

    A combination 3-way USB hub and N-way card reader. With a very short USB A to mini-B cable. The hub feature is not so useful, but the card reader often gets used for reading random people’s memory cards so I can copy their photos.

    A MacBook VGA monitor cable dongle thingy.

    Some more keys.

    Another USB memory stick. Not modular, 128 MB of soldered memory.

    Charger for Canon digital camera battery.

    Charger for Nintendo DS. The last two items are slightly amusing, as the neither the camera nor the DS is in the bag.

    What’s in your bag? Are you going to tidy it out / stock it up for the conference season?

    OS X and X11


    Apparently Matlab on the Mac is still in the stone-age of using X11, and it sucks.

    Yes, Apple provide X11 with OS X. But you’re not supposed to use it. It’s not for lazy retards to base their Mac platform strategy on, it’s for desperate people who absolutely must use that X11 app that some scientist wrote back in 1993.

    I find mathwork’s attitude a bit whiny. On moving away from X11: “this is a multi-year endeavor”. Well, yeah. I’m sure it is. But OS X has been out for over 8 years now!

    Bottom line: if your answer to “do you have a Mac product” is “Yes, we use X11”, then you do not have a Mac product. You have an inexcusable pile of dingo turd. If the answer is “Yes, it’s written in Java” then you just upgraded your turd to offal. Congratulations.

    Of course, this is just my opinion. My opinion and the opinion of every other Mac user. If we liked using X11, we’d all be using Linux already.

    PS: Firefox gets it right. How hard can it be?

    What Good is PyPNG? (and some MDIS pictures of Earth)


    I originally wrote PyPNG so that I could manipulate PNG images using Python. My original itch was a tiling application. Since then I’ve been using to develop all sort of little image utilities and investigate PNGs. pipdither is written in Python and the initial version took about an afternoon of fairly lazy coding.

    Python is a great language, and now that I’ve got a module for creating PNGs, it’s easy to create other image tools in Python.

    Here is a link to an image of Earth taken from the Messenger spacecraft (a 2.1e6 octet file that you won’t be able to read). Messenger was on an Earth flyby on its way to map mercury when it took the image. The image is archived by NASA’s PDS and stored in its IMG format. Which is basically a whole boatload of metadata followed by the image data in row major order at 16-bits per pixel.

    Because it’s basically just a bunch of numbers after the metadata, this is really easy to grok using Python. Here it is converted to PNG using PyPNG (this is a small 8-bit version, follow the link to get the full 12-bit version):


    It looks rubbish. Someone has turned the brightness up too high on the telly. It looks rubbish for two reasons: a) I embedded a gAMA chunk to specify a gamma of 1.0 (linear); and, b) the camera’s black-level does not correspond to a code of 0. Here’s a histogram:

    Sorry the histogram is a bit rubbish (the image is 12-bit, so the histogram is 4096 pixels wide). But you should be able to see that the first chunk of codes (on the left) are unused. That’s because even though there’s no light falling on the sensor, there’s thermal noise or some residual current drain, or something. So picture black is represented by a code of about 7%. And because I specified a gamma of 1.0 that 7% value in the PNG file actually looks not very dark at all.

    Actually I had to fiddle with wordpress quite a lot to get it looking this bad. If left to its own devices wordpress will create its own scaled down version of the image and it THROWS AWAY the gAMA chunk. If your computer ignores the PNG’s gamma chunk then maybe the image doesn’t look to bad to you (a grey level of about 7% in some sort of perceptual space looks fairly black, not actually black, but good enough that you probably won’t tell unless you open a black image next to it).

    So clearly to get a good image I should choose some black-level, 7% say, subtract 7% from all the pixel values and rescale to fit the full range. Yeah I could. But that would destroy the original data and impose my own processing on it. What if you wanted to do your own black-level compensation?

    I had an idea. I could create an ICC profile that clamps the bottom 7% codes to black, then embed the profile in the PNG. The profile looks like this (in Apple’s ColorSync utility):

    Apologies (yet again) for the fact that the screenshot doesn’t fit. WordPress would’ve scaled it, but apparently it falls over when trying to scale a 4-channel PNG. The screenshot has an alpha channel because it’s a shot of a Window and the window has chamfered corners which require alpha. How amusing.

    Anyway. Input is on the x-axis, output is on the y-axis. You can see that this TRC curve maps the bottom 7% into 0, and the rest goes linearly.

    This being the first time I’ve deliberately embedded a colour profile into a PNG image, I’m slightly gobsmacked that it actually works:


    You get the best of both worlds. A plausible looking picture for viewing, and all the original image data if you want to dink around with it yourself.

    I’m pretty sure that a gamma of 1 is correct, by the way, because the values in the IMG file are raw 12-bit sensor values and they generally respond linearly to light. Actually they’re not quite raw, the onboard computer has compressed the image before it was transmitted to Earth, you can see compression artefacts in the “noise” in the dark regions of the picture (if you look at the version that doesn’t have a colour profile).

    PS If these images look the same, your computer isn’t doing colour collection. It sucks. It works on a Mac [ed: if you use Safari]. Try my later article to see a simulation of what it looks like when it works.