Unusable Train Reservations

Returning from EuroPython I got the 1730 from Birmingham New Street to Sheffield. The seat reservations in this class of train appear above each pair of seats on a little illuminated dot matrix display that shows 2 rows of text. Each row displays 16 characters (it was quite tricky to count, but it was close enough to 16 that surely, if there is any god, it must be 16).

Each row displays the seat reservation information for one seat. As a little message that scrolls along. Most of these messages were of the form: “20 This seat is not reserved”. This message is 28 characters long. Which means it needs to scroll to fit on the display. Some genius decided to pad the scrolling message with 16 blanks, so that the end of the message scrolls off completely before the message begins again. So, for the display of this message, there are 44 states that a row can be in. Each state corresponds to a position in the 44 character string (each position in the string can be identified with the display state that has that position at the extreme left-hand end of the display).

The seat number is 2 digits long. It is only displayed for 15 of the 44 states. Meaning it is only visible for 34% of the time. It’s actually kind of important to display the seat number. Especially as they made the mistake of putting the larger of the two seat numbers on the top row. The two rows display the reservations for a pair of seats: N (bottom row), and N+1 (top row). Nuts. In fact it’s not necessary to display the seat number on the display itself. Adjacent to the display (on either side) are stickers showing the numbers for the two seats and whether they are window or aisle. It would be a trivial design change for these stickers to point to the appropriate row of the display.

Some of the time the display will show t reserved or eserved or something else that could reasonably be mistaken for reserved. So if you just glance at the display then you have about a 7% of interpreting it as “reserved” (when in fact it’s not reserved).

The shorter the message is, the more likely we are too be able to comprehend it instantaneously without having to wait for it to scroll. So it would be good to get rid of unnecessary text. “This seat is” is totally unnecessary. We’re on a train, we can tell that the little display refers to a seat reservation. So all it needs to say is “20 not reserved”. And that’s only 15 characters long, so it can be displayed permanently, without needing any anti-assistive scrolling.

The case where the seat is not reserved is a bit special, but it’s worth concentrating on, because those are the seats that people will want to find when they are boarding the train. At least, people without reservations. People with reservations don’t need the overhead displays at all because they can just look at their ticket to find the seat number. To recap: The reservation displays are only useful for people without reservations, so they should be organised around making it clear which seats are free.

The remaining cases, where the seat is reserved for some of the remaining journey, should probably be handled with text that is something like “free until chesterfield” or “reserved until york”. Possibly the “free” and “reserved” can be “stuck” at the left-hand side of the display while the remainder of the display scrolls to show the whole message. Dunno. But I bet a day of trying out a dozen ideas would be a vast usability improvement on how it works now.

After Copper

We gather in the dim space afforded by the overgrowing ash (Fraxinus excelsior) and cow parsley (Anthriscus sylvestris). The trees provide some shelter from the rain, after all, it is July in the Peak District. The key is brought forth. The old iron gate swings open. We step down into a small stream, and through the gate, passing under a stone reading “DEEP ECTON: DRIVEN 1774″. A tunnel arch leads inward, towards the heart of the hillside.

My wellingtons are in muddy water. The pavement beneath, which I cannot see, is uneven. The daylight fades, but my eyes have not yet adapted to the dim torchlight. Stumbling forwards, reaching out for the walls. Tracing my hand along the wall to steady myself, missing my footing as my hand discovers crumbling gaps in the walls. The dwarves that once worked this place left over 100 years ago; they must have been dwarves, for sure they were smaller folk than me, for I keep banging my helmet on the odd out of place stone in the ceiling. We are in Deep Ecton Mine, once the source of the Peak District’s copper.

After a short while, plaster gives way to (limestone) brick. Little nooks appear, presumably once filled with candles. Then the brick gives way to a rough unlined tunnel through the bedrock. The horizontal tunnel we are walking, stumbling, along would have been called an adit by the dwarves. Its upward slope is imperceptible save for the fact that we are walking through a stream; running downwards and outward to rejoin its brethren waters of the Manifold. My eyes are adapted now, and my foot and inner ear give me mostly steady passage.

We pause at the first “chamber”, a swelling of the rough tunnel. A crawl above a pile of rock debris, “deads” as the dwarves called them, leads to a much older, smaller, adit. Its roof has fallen in, now blocked and unsafe. A narrow shaft leads upwards, presumably once connecting to the surface. The surface, and all evidence of the outside world, seems very distant now.

Passages leading off into the gloom. Stepping over discarded ironwork, occasional rocks (fallen from the roof?), and… tramway sleepers. The adit would once have been busy with minecarts. I see a little metal tag marked with “A6″ (a modern survey tag). One dwarf has mezzotinted his initials, IB, into the wall with his pick. The whole thing reminds me of, well, what else but Colossal Cave (and damnit, why didn’t I think to try saying “PLUGH”?).

The “IB” chamber has a big rectangular hole in the floor. Full of water. We are on the lowest dry level of the mine (the entrance that we used is only a few meters above the River Manifold). Just around the corner we come to the main attraction. A huge vertical cavern, or pipe. Big enough to contain a house and extending upwards in a complex series of natural shafts, platforms, and other connecting chambers. The house would have to float, for the “floor” of the cavern is a gigantic pool filled with crystal clear water. No plant infiltrates, and no animal stirs the mud. A tiny waterfall chirps and trickles its way down into the pool from some higher cavern. Downwards, through the water, we can see that the pipe continues. Occasionally we can see massive wooden props, essentially whole mature oaks trimmed to a rough rectangle, fitted across where they once would’ve supported platforms. The mine continue downward, as does the pipe, for at least another 300m. But the underwater areas remain unexplored since a diver’s death here in the 1960’s.

This pipe is where the copper was. Formed when the bedrock flexed and cracked, allowing copper carrying water to seep in and deposit its lode. Of course the dwarves took all the copper, but we can see occasional spots where copper has remineralised and formed a greenish colouration on the walls (copper carbonate?). All around we can see places where the bedrock has bent into huge curves, cracked, and the cracks filled with worthless calcite (by contrast, the limestone bedrock nearer the entrace has no calcite veining).

This pipe was formed by hacking out valuable copper ores. The next cavern is 10 metres across and big enough to stand in (and in parts up to about 8m high). It was dug out not for copper, for it never contained any, but to house an engine. It is the engine chamber. Housing engines for pumping water out of the depths. Engines powered not by steam, but by horse, and by water. The amount of effort involved is quite incredible: all that worthless rock removed to create this large room, rivers diverted, engines installed, a vertical 300m oak beam (bolted in sections, naturally). All just to remove water so that the lower sections of the mine could be worked for their copper.

Then the tour was over. We had to make our way to the endgame before our batteries ran out and the cave collapsed (just kidding, another text adventure reference). Whilst we had been underground for quite a while, it didn’t take us long to go back along the adit and reach daylight and the smell of fresh air, a smell you really appreciate when you’ve been underground.

Thanks to the staff of the Peak District National Park who used their own time to give us the opportunity to see the mine and benefit from their experience.

My EuroPython talk: Python Sucks!

On 2009-07-01 I gave a talk at EuroPython called “Python Sucks!”. They made me change the title of the talk, but the first slide sticks, so “Python Sucks!” it is.

It was a bit of a misleading title. As I did actually mention some things that I like about Python.

The slides (updated in blue to add useful things that people said in the talk itself). are available in PDF. I’m not sure the slides are particularly useful without a transcript; it’s not always clear if the point illustrated on the slide is something that I think is a good thing, or a bad thing.

I was a bit overwhelmed with the talk actually. I was thinking that, as I am not as famous as Bruce Eckel or Tony Hoare, about 30 people would turn up; and I think I could probably wrangle about that many people. The Recital Hall holds about 150 people, and it was pretty much full. *gulp*.

The audience included Tony Hoare (Man of Science); when I spotted that I sort of thought “oh no, Tony Hoare is in the house, I’d better behave now”. He usefully (and somewhat embarrassingly on my part) suggested that Occam be added to the cloud of “languages I don’t know enough about”. And it should.

One of the points of the talk was to get the audience talking; I think I did okay at providing a very lively forum for people, not just me, to get their python gripes off their chest. The contributions from members of the audience were well appreciated, and often informative. Certainly I felt that the audience provided a useful contribution, which of course made my job easier.

Later on in his keynote Tony Hoare said something like “from what I’ve seen here today you are doing a good job” [of being scientific engineers, or of steering Python, or something]. I hope he wasn’t referring to me.

Note to self: do not show a slide with “distutils” on it to 150 people. Unless you have nothing to say.

Tony Hoare, man of Science

At EuroPython, on 2009-07-01, Tony Hoare gave a very interesting speech on the opposition between science and engineering. In many ways it reprised the themes of Laura Creighton’s keynote from PyCon UK 2007, but from a science perspective rather than a history of science perspective. In the middle of the talk, some wit twittered “Stand back! Sir Tony Hoare is about to do science!”; shame it wasn’t on the big twitter wall.

Tony Hoare is clearly old skool. His slides had the calm and aged patina of the OHP era, and I thought they were all the better for that. If you have a message, then that message can be conveyed without all the flash and shine that PowerPoint tempts you with (although, being a Microsoft man, of course his slides were in PowerPoint). As Andrew Kuchling says: “(good talk, plain slides) > (bad talk, fancy slides)”

I was particularly impressed with this slide from Tony’s talk outlining a few “special interests” of the scientist and engineer respectively:

On the scales on the side of Science we have things like “long-term”, “perfection”, and “originality”. Balancing the scales for Engineering we have “short-term”, “adequacy”, and “best practice”.

What I liked about this slide is that many of the things on the Science side would be seen as defects in an engineer, and many things on the Engineering side would be seen as defects in a scientist. We have all seen scientists attacked for relying on intuition or merely amalgamating best practice. And what engineer has not been barracked (by their manager) for attempting solutions that were too perfectionist or wasting time on long-term goals?

Tony Hoare’s insights are clearly the product of long and hard work. He seems very optimistic about the possibilities of a virtuous feedback between the engineering and scientific sides of computing. Perhaps we have every right to look forward to the day when “Software will be the most reliable component of every product which contains it.” (the last slide from his talk). But right now… it seems a long way off.

Python: dictionary of functions

On twitter recently I was wondering what the best way to create a dictionary of functions in Python was. On a suggestion of Paul Hankin’s I looked into classes and metaclasses. The most direct way I discovered is to use a class; metaclass not required:

class foo:
  def bar(): pass
  def zon(x): return 1+x
fundict = foo.__dict__

This is direct, but not at all obvious. Hint to people that are mystified already: the __dict__ attribute of the class is a dictionary containing everything defined in the class.

fundict is now a dictionary that contains the two functions bar and zon.

I made this discovery a few days before EuroPython, and was fortunate enough to bump into Bruce Eckel in the hallway at EuroPython just before he gave his metaprogramming talk, so I showed it to him in what I call my “naughty decorator” form:

def asdict(x):
  return x.__dict__

@asdict
class baz:
  def bar(): pass
  def zon(x): return 1+x

(By the way, one of the things I learnt at EuroPython is that the crappy decorator syntax gives Java refugees a warm fuzzy feeling).

Bruce pronounced this “not as naughty as you imply” and “worth showing” (this blog post was half-written before I bumped into Bruce, but he is definitely encouraging me).

It’s worth showing for the following reason:

The functions you get from the class’s __dict__ dictionary are not the same as the methods you get by accessing the attributes of the class. In other words «foo.bar is not foo.__dict__['bar']». I was surprised by this, and so was Bruce.

As well being a little weird, it makes a compact example to show the different between a function, an unbound method, and a bound method.

I hope I don’t need to introduce a function. It’s just a thing you call with some arguments. Where it differs from a method is that a method is regarded as a message sent to an object, and receives that object as its first argument.

In Python, an unbound method is a method that isn’t associated with any particular object; it requires an object as its first argument (and Python loses big here, by requiring the object to be an instance of the class that defined the method). A bound method is a method already associated with an object; that object becomes the method’s first argument when the bound method is invoked with the remaining arguments.

So if we define a class foo (as we did, above), then:

foo.bar is an unbound method;

foo().bar is a bound method (bound to the object we just created by invoking the foo class); and,

foo.__dict__['bar'] is a function.

This last fact was a great surprise to me. I had expected it to be an unbound method, and thought that my naughty decorator would have to have some hacky code to dig the function out of the unbound method. But it doesn’t.

Tiny problem: Using the asdict gives a dictionary that contains __module__ and __doc__ keys. Solution: another decorator:

def cleandict(x):
  for k in ('__module__', '__doc__'):
    del x[k]
  return x

@cleandict
@asdict
class baz:
  def bar(): pass
  def zon(x): return 1+x

EuroPython 2009

Overall I had a really enjoyable time, met lots of interesting people, some new and some renewed friendships, and learned some good stuff.

Too many web frameworks. Too many VMs.

What I like is the diverse range of applications to which people put Python. The Guardian use it so we can inspect our MP’s expenses. Gregor Lindl uses it and Papert’s turtle graphics to teach. The Deutsches Zentrum für Luft- und Raumfahrt use it to manage the extraction and shuffling of petabyte (that’s 10**15!) datasets across the supercomputing networks used by climate scientists. The talented Stani Michiels creates the new Dutch Euro coin designs using Python. Weather trading derivatives. Video workflows. Collaborative mathematics. Games. Academic document archives. Billing. System Administration. The list goes on.

Naturally I managed not to go to most of those talks (apart from keynotes (like Tony Hoares) and lightnings, I went to 2 talks). That’s partly because going to 4 or 5 talks a day (which would easily have been possible given the packed schedule) makes you dead tired and causes leaky brain; partly because there were timetable clashes! But mostly because I was writing the materials for my two talks.

Writing your talk at the conference itself is… exciting. And intense. And it probably gives the conference organisers the willies (as in, “where are your slides?”). It did mean that I was able to include some stuff from the conference itself in my talks. A war story I picked up in hallway chat about having to use 6 year old versions of Python on a government IT project made it into my “Loving Old Versions”" talk. Dr Sue Black’s talk about Bletchley Park was in the timetable, and that prompted me to put a reference to Turing’s coffee mug into my “Python Sucks!” talk (Sue Black got there first with the same anecdote; ah well). The “Python Sucks” talk also got a couple of references to Bruce Eckel’s keynote which covered some of the same ground (just using a lot more space, according to Thomas Guest).

But it’s way stressful (Rob Collins’s massage to raise PSF funds was very helpful in that regard). Next year I’ll leave the laptop at home, and give a presentation using a marker pen (well, I will if they accept it!).

Convert Plan 9 images to PNG

Should anyone be in the slightly unusual position of wanting to convert a Plan 9 image to PNG format, they can now use the plan9topng utility in PyPNG:

curl http://plan9.bell-labs.com/sources/plan9/sys/games/lib/sokoban/images/right.bit |
./plan9topng.py > glenda.png

Glenda says “hi!”:

Python: Some Naughty Features

Things in Python that I mostly like, but make me feel a bit naughty when I use them. I air my smelly code as a sort of cleansing ritual and perhaps to make you feel a little bit better about your own dirty habits.

zip(*[iter(s)]*n)

I now tend to call this zip/iter thing group. It came to me in a dream.

itertools.chain(*s)

The “inverse” of the above. I only “discovered” this recently. A good couple of months after the above discovery.

Bound methods of simple values. Example: In order to convert a sequence of image samples from perceptual space to linear space we need to raise each one to the power 2.2:

map((2.2).__rpow__, mylist)

«(2.2).__rpow__» is equivalent to the following function:

def f(x): return x**2.2

(as a lambda: «lambda x: x**2.2»)

__rpow__ is the swopped version of __pow__. Using «(2.2).__pow__» would be equivalent to «2.2**x», which isn’t what I want at all.

Literal string concatenation. To be honest, I’ve no idea why this feels a bit naughty to me, but it does. C has it, so why not in Python? Perhaps it’s because I often end up using it in cases where there are other problems with the code (this example from PyPNG):

    raise FormatError("Illegal combination of bit depth (%d)"
      " and colour type (%d)."
      " See http://www.w3.org/TR/2003/REC-PNG-20031110/#table111 ."
      % (self.bitdepth, self.color_type))

Sometimes I get a bit OCD about this and start removing extraneous string concatenation «+» operators from other people’s code.

The split thing that I mentioned in an earlier article. Here’s an example from Clear Climate Code:

noaa = """
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.temperature.inv
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/hcn_doe_mean_data.Z
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/station.inventory.Z
""".split()

It just seems so much nicer to be able to add a URL to the list without having to add any more syntax (and in this case, I think I originally created the list by pasting it in from a text file, so using split meant less reformatting).

Doing non-trivial things in a module’s top-level. Another example from PyPNG:

_signature = struct.pack('8B', 137, 80, 78, 71, 13, 10, 26, 10)

This is just a complicated way of writing an 8-byte literal string. It’s written this way so that the crucial 8 bytes are visible as decimal values, and can therefore be checked more easily against the relevant part of the PNG standard which also uses a decimal notation.

Passing a function with side-effects into map. pipcolour (an undocumented utility in PyPNG) counts the number of colours used in an image. In the code below, pixels is an iterator that yields each row of a PNG image; the pixels from each row are added to a pre-existing set of colours. Like this:

    col = set()
    for row in pixels:
        # Ewgh, side effects on col
        map(col.add, png.group(row, planes))

(in this code png.group is the grouping function which is the first item in this article; for an RGB image, planes is 3)

Now, it strikes me that I could’ve gone:

set(group(itertools.chain(*pixels), planes))

but that has radically different memory usage. The first form only loads one row at a time into its working set; the second form loads the entire decompressed PNG image into the working set (it’s all the fault of «*pixels»). If I was daring enough to require Python 2.6 then I could’ve used itertools.chain.from_iterable, but that feels clumsy. It one of those cases where I feel that Python is unnecessarily forcing me to choose between clarity and efficiency.

I think the above also illustrates a weakness with the sort of delayed evaluation programming that you get when using iterators: Apparently simple transformations to the code can result in huge changes to the space/time complexity. This is not a weakness that is unique to Python, any style of delayed evaluation has this risk (Haskell’s lazy evaluation, for example). When I was learning Prolog my chief criticism of it (and I think it’s still valid) was that it seemed to be fairly easy to be sure that your program would compute the correct answer, but very difficult to be sure that the answer would appear before the heat death of the universe.

Violating PEP 8 so I can write one-line functions:

    def test_gradient_horizontal_lr(x, y): return x
    def test_gradient_horizontal_rl(x, y): return 1-x
    def test_gradient_vertical_tb(x, y): return y
    def test_gradient_vertical_bt(x, y): return 1-y

Did you spot I did one of those earlier? How naughty. Of course PEP 8 kind of approves of this, because “a foolish consistency is the hobgoblin of little minds” which you probably know from PEP 8 but is of course from Ralph Waldo Emerson. In a curious coincidence I just happen to have a copy of his “Self-Reliance” on my desk; and I checked, he really did say that. In fact, the fuller quote is much more poetic than I was expecting: “A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines.”.

Indexing with a boolean expression. Yet another example comes from PyPNG and illustrates how well written it is. Say you want to create an array of pixel values or you want to pack them into a string (using struct). If the number of bits per value (the bitdepth) is <= 8 then you want to use «array('B')», otherwise you want «array('H')». Select the format like this:

fmt = 'BH'[self.bitdepth > 8]

Iverson’s Convention for the win! (and it’s better than PEP 308 because it works on Python 2.4)

Now, this is naughty: Creating a naughty bit of Python code merely for purposes of this blog article. Well, that’s not quite true, I got distracted by some Python code and that code and a stupid idea I had a few days ago clicked together. The code is witty, but should never be used. PyPNG allows the caller to specify the zlib compression level as an optional argument when creating a png.Writer object. This is the argument to the zlib.compressobj constructor. The default (for PyPNG) is to use zlib’s default. Now it’s clearly not the business of PyPNG to know what zlib’s default is, so I don’t want that appearing in the code. So the code looks like this:

if self.compression is not None:
    compressor = zlib.compressobj(self.compression)
else:
    compressor = zlib.compressobj()

Now, what I really want is a way to say «pass in x as an argument, but only if it’s non None». If I had some expression, E, that evaluated to a list that was either the singleton, «[x]», or the empty list «[]» then I could get rid of the if and go:

compressor = zlib.compressobj(*E)

«*E» would expand to either the empty argument list, or an argument list with just one argument.

I discovered an E: [x for x in [self.compression] if x is not None]. So now the code is:

compressor = zlib.compressobj(*[x for x in [self.compression] if x is not None])

Well, the code isn’t really that, I’m not that much of a monster to check that into source control.

Multiplication, Addition, Counting, and Python

Jason Dyer’s teasing post got me thinking. About how Python could be used to give some insight into the meta-cognitive aspects of whole number multiplication. Natch.

When children solve a multiplication problem by correspondence, the objects in the multiplier set are mapped over for each object in the multiplicand set (hmm, or is it the other way around?). A typical procedure for multiplying 4 cakes by a price of £2 per cake might be to point to a cake with the left hand and then count up the 2 pounds using the right hand, then move the left hand to the next cake and repeat the count with the right hand, with the oral count continuing up; this is repeated until the left hand has exhausted all cakes.

We can model this in Python with the following program:

def mul(multiplicand, multiplier):
    count = 0
    for i in range(multiplicand):
        for j in range(multiplier):
            count += 1
    return count

Whereas children using repeated addition do something more like this:

def mul(multiplicand, multiplier):
    count = 0
    for i in range(multiplier):
        count = add(count, multiplicand)
    return count

In this case, add is a subroutine. You could define one easily enough in Python or else you could go «from operator import add».

Clearly the second program is more efficient than the first, both as a Python program and as a manual procedure performed by children; it’s more the latter that I’m interested in, I’m using Python to describe the procedures. However, the second procedure requires that add is already adopted as an internal procedure. Of particular note is that, apart from count, the second procedure uses only one state variable, i; the first procedure uses two.

At the stage where multiplication is introduced, many children will not yet be performing addition accurately without counting. Effectively add is not yet available to them as an internal procedure. This is likely to be a problem because this type of learner is unlikely to be able to accurately add the multiplicand to the count without getting confused about where in the multiplication procedure they were.

As an example of what might go wrong, imagine that a learner starts by using the left hand to maintain the i variable in the second procedure (above); this hand will count from 1 to multiplier (hmm, there’s a small sleight of hand going on here, the Python counts from 0 to n-1 whereas most learners will prefer count from 1 to n). The count will be maintained orally (in other words by speaking out the successive multiples of the multiplicand). Begin by raising one finger of the left hand and uttering the multiplicand (the initial count). Now we need to add the multiplicand to the oral count. Maybe the learner can do that without using their fingers, maybe they can’t; in any case depending on the parameters chosen, at some point some learners may need to use their hands to perform the addition. So then the procedure for addition kicks in as the learner adds the multiplicand to the current count. Many learners will have personal procedure for addition that requires both hands. The addition may be performed accurately, but the i state variable will be lost. We lose track of where we were in the multiply procedure.

If the learner can accurately add the multiplicand to the count mentally, then they stand a much better chance of performing the second procedure. This is what I mean by having add available as an internal procedure.

The first procedure can be thought of as a way of simultaneously arranging to perform the multiplication and the additions required by the multiplication without having any state variables overlap. Thereby minimising the chance that confusion will result. Most learners will be capable of keeping track of the required state variables to perform the multiplication, but if left to their own devices may choose methods where the state variables overlap (in other words, they run out of hands). Thus, they can benefit by being guided towards a procedure which they can manage.

Another way to think about this is that at the sort of age where children begin to learn to multiply, their procedure for addition is leaky. It is not fully abstracted; performing addition may require the use of hands (for state variables), making the addition leak over any other use of the same hands to maintain state.

It seems to me that only when a learner can perform a typical count + multiplicand addition accurately in their head are they ready to perform multiplication as repeated addition.

Oh yeah, the original research on the whole “multiplication is/ain’t repeated addition” debate. It sucks. They test children at times t0 and t1 with a randomly chosen math related intervention in between. It seems to me that a more carefully designed study should have also included a non math-related intervention such as giving the subjects fish-oil pills or teaching them to conga. After all, if I was being tested on one day and then told that I was going to sit a similar test tomorrow, I would bone up on the material before the second test, regardless of what I was being taught. Wouldn’t you? They make no attempt to account for this effect.

Appendix for Python hackers

The first definition of mul that I give is of course completely worthless as a practical implementation in Python. However, note the following: «10*10L» is 100L but «mul(10,10L)» is 100; in other words mul returns an int if it possibly can.

Anonymous things in Python

This post of pydanny prompted me to flesh this article out (previously it had just been floating around in my mind). pydanny’s article reminded me of when I first learned about lambda and how scary it was. That was in about 1988 when I was trying to get to grips with XLISP on the Atari ST (hah! a Lisp dialect so old it still spells its name with capital letters. sweet). Previous to XLISP I had been programming in (Sinclair) BASIC, z80 machine code, and 68000 assembler. Oh, and FORTH. Maybe you can imagine how scary and alien a concept like lambda was. I can just barely remember how confusing it was. I don’t think the confusion really passed until I started dabbling with ML, in 1991. By the time I started dabbling with the lambda calculus in 1993 I was actually pretty comfortable with it. So, only 5 years of exposure to get lambda to get comfy.

Of course now, I know that:

A lambda is a simply function with no name.

I emphasise the simply because I feel that that’s where the problem of understanding lay (for me). In BASIC and FORTH it was impossible to have a function without a name. In fact, the very thought was literally inconceivable. At least for a few years. Even in machine code where functions don’t have names they do have concrete locations, and they have names in the listing.

It wasn’t until many years later, in fact perhaps relatively recently, that I realised that there was a sort of hierarchy of things that must be named, and things that need not be named. For example, an expression, like «a**2 + b**2 + c**2» is an anonymous value, a value that does not need a name. More importantly, so are all the sub-expressions inside that expression. What would a language look like if it didn’t have anonymous values? Well, something like this:

a2 = a**2
b2 = b**2
c2 = c**2
s = a2+b2
s = s + c2

Do you see? Each expression must be given a name (by assigning it to a named variable), and no expression can contain a sub-expression, because that sub-expression would be anonymous.

Programming in assembler is a bit like this. The names are the registers and every expression has to go in a register. If you run out, you have to rename everything by spilling to the stack. It’s partly why it can be so tedious.

So almost everything worthy of being called a language has anonymous values. At least simple values, like numbers. In the BASIC era it was common for strings to be named only. The kinds of restrictions you had to deal with were things like not being able to create a string from part of an existing string. You had to first create a named variable that was the target string, then copy part of the source string into the target string. Maybe you had some string expressions but there were restrictions on where you could use them. For example maybe you could only pass named strings to functions, so you could go «T$=MID$(A$,12,16):PROCFOO(T$)» but not pass the MID$ expression directly: «PROCFOO(MID$(A$,12,16))» (bad bad bad). Life was hard, and we licked coal for breakfast.

Arrays were another thing where you had to name them and couldn’t pass them to functions (for example). AWK is still like this (arrays cannot be returned from a function, for example).

Speaking of arrays, an element of an array is a bit like an anonymous variable:

a[i] = x

Above, a[i] specifies the place where the computed value, x is this case, goes. I say “a bit like” because you could equally well think of the element being named by the array name and an small integer as a pair. A place where we can store a value that changes is called a variable. You knew that right? It’s just that most people think of named variables when you say variable. The generalised term for “anything you can put on the left of an assignment” is lvalue. A term you mostly see in the C programming language but often elsewhere too. Common Lisp calls such things, places. How sensible.

So the hierarchy of things you need not name starts something like:

simple numeric expressions

strings

arrays (including hash-tables, and so on)

objects (incuding structs, records, pairs, and so on)

functions

places (actually, not sure where this should go really)

As you go down the list, you see fewer and fewer languages that give you anonymous versions of the thing in question. Although these days we probably all agree that any decent language will have anonymous versions of all of the above. Python does, and that’s a Good Thing.

One reason for arranging things this way is that we can use it to both explain and motivate things like lambdas. A lambda is an anonymous function, and you remember how useful it was when you realised you could have string expressions and you didn’t have to store intermediate strings in variables? You could just manipulate strings without having to name them? Well, the same is true of functions.

So what else could and should be anonymous? Well we can look to other languages for inspiration. ML has anonymous types. Java has anonymous classes. So perhaps the list can continue:

classes

types

(note that a class is a sort of type, so this order has a natural feel to it).

So everything I said about anonymous functions applies to anonymous classes and anonymous types. They’re great, and every language needs them. Basically my philosophy here is that if there’s a thing in a language and you can name it, then you should also be able to have anonymous versions of those things. That will make the language better.

So what other sorts of things does Python have that we haven’t removed the names from yet? Modules. For a long while I wished that Python had anonymous modules, a way of getting hold of a module without it polluting your namespace. Mostly I just wished for this because it would make Python more orthogonal, or better as I like to say. Of course modules in Python are first class citizens. Once you’ve imported the module struct, then struct refers to the module as a first class value, and it can be passed around and so on. That’s how help(struct) works. Recently I found both the way to get at modules anonymously (without importing them into a namespace) , and a reason why you might want to.

I’m a tidy sort of guy so when I program in Python I put my imports only in the functions that need them (a habit I picked up from Command Line Warriors but buggered if I can find the original inspiration). Well, sometimes I am.

I often find myself debugging my Python by the time honoured tradition of inserting print statements. Because of my tidy habit, I quite often find that the sys module is not in scope (in fact, often the only function that has «import sys» is main. So this fails:

print >>sys.stderr, "length:", l, "data:" x

And I’ll be damned if I’m going to add an import sys and forget to remove it later. But this works:

print >>__import__('sys').stderr, "length:", l, "data:" x

Not a very nice syntax though, is it?