Explaining p += q in Python

2013-10-29

If you’re a Python programmer you should know about the augmented assignment statements:


i += 1

This adds one to i. There is a whole host of these augmented operators (-=, *=, /=, %= etc).

Aside: I used to call these assignment operators which is the C terminology, but in Python assignment is a statement, not an expression (yay!): you can’t go while i -= 1 (and this is a Good Thing).

An augmented assignment like i += 1 is often described as being the same as i = i + 1, except that i can be a complicated expression and it will not be evaluated twice.

As Julian Todd pointed out to me (a couple of years ago now), this is not quite right when it comes to lists.

Recall that when p and q are lists, p + q is a fresh list that is neither p nor q:


>>> p = [1]
>>> q = [2]
>>> r = p + q
>>> r
[1, 2]
>>> r is p
False
>>> r is q
False

So p += q should be the same as p = p + q, which creates a fresh list and assigns a reference to it to p, right?

No. It’s a little bit tricky to convince yourself of this fact; you have to keep a second reference to the original p (called op below):


>>> p = [1]
>>> op = p
>>> p += [2]
>>> p
[1, 2]
>>> op
[1, 2]
>>> p is op
True

Here it is in pictures:
before.dot
fresh.dot
augment.dot

Because of this, it’s slightly harder to explain how the += assignment statement behaves. For numbers we can explain it by breaking it down into a + operator and an assignment, but for lists this explanation fails because it doesn’t explain how p (in our p += q example) retains its identity (the curious will have already found out that+= is implemented by calling the __iadd__ method of p).

What about tuples?

When p and q are tuples the behaviour of += is more like numbers than lists. A fresh tuple is created. It has to be, since you can’t mutate a tuple.

This kind of issue, the difference between creating a fresh object and mutating an existing one, lies at the heart of understanding the P languages (perl, Python, PHP, Ruby, JavaScript).

The keen may wish to fill in this table:

p q p + q p += q
list list fresh p mutated
tuple tuple fresh fresh
list tuple ? ?
tuple list ? ?
About these ads

2 Responses to “Explaining p += q in Python”

  1. phil Says:

    As you allude to, this is almost certainly an artifact of the implementation of __iadd__ for lists; presumably it’s calling p.extend(q) and returning p. What different collection types do will depend on their implementation.

    You’d think the language gatekeepers would prohibit implicit side-effects in operators, at least in the standard library. But given GvR’s famous aversion to FP in Python, perhaps we shouldn’t be surprised. After all, it’s hardly the first time Python’s language designers didn’t think things all the way through.

  2. g Says:

    phil,

    It seems pretty clear from PEP 203 that the fact that += operates in place on lists (mutating the LHS object) is very much deliberate and that the ability to do this sort of thing is a large part of why operations like += exist at all in Python.

    Whatever may be said for or against it, it isn’t quite a matter of the designers not having thought things all the way through.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: