I learn Python

2008-09-25

It takes a long time to get to the point where you stop learning a language, perhaps you never do stop.

Some useful Python tidbits I’ve recently picked up that I feel I ought to have known much earlier:

Using the dict constructor instead of the literal syntax:

# awkward
{'cht': 'lc', 'chs': '500x600'}
# nice
dict(cht='lc', chs='500x600')

This is mentioned in the tutorial. Bad code monk. (Ah, this is that new fangled Python 2.3; that explains why I previously didn’t know it. Basically I learnt the vast majority of my working Python when lambda was new and «from __future__ import nested_scopes» was necessary)

When I met Raymond Hettinger at PyCon UK (I was doing a “Code Clinic” with him) one of the first questions I asked him was “when are we going to get unzip“; when he replied “zip is also unzip, you can just use zip(*l)” I had one of those forehead slapping moments (quite literally I think). I course I knew that “zip(*l)” transposed a matrix (see Norvig’s IAQ), but I wasn’t used to thinking of a list of pairs, say, as being a matrix that I could transpose to get a pair of lists. Kick! Punch! It’s all in the mind.

Using split instead of literal lists:

# awkward
['F', 'PD', 'AD', 'D', 'TD', 'ED', 'ABO']
# nice
'F PD AD D TD ED ABO'.split()

This isn’t so much a new trick, but I was always a bit embarrassed about it. But now I’ve seen other Python programmers do it too, so I know it’s more socially acceptable.

Now one that I don’t actually use so much, nicked from Thomas Guest’s blog:

"chd=s:%(xs)s,%(ys)s" % locals()

Can you see what’s going on? The variables xs and ys are local variables whose values are spread into the string using «% locals()» as a sort of “interpolate from local variables” operator. I’m not a great fan, but I can see that it is useful. I’m not such a fan because it uses locals which is very cool, but could interfere with a compiler’s optimisations. Though in this case, with a constant string on the left of the % operator, the compiler has enough information to do a good analysis.

Speaking of Python string formatting, I was disappointed to learn that the optimisation of compiling format strings, which is routine in the Lisp world, is not generally done in Python. The optimisation I am talking about is one where «”some constant string” % stuff» gets converted into «some_function(stuff)» where some_function is a compiled function that does the formatting. It’s just one of the signs of how immature whole Python thing is.

And one specially for Pythonistas doing Google charts:

>>> d = dict(cht='lc', chs='500x600') # The dict from above
>>> '&'.join(map('='.join, d.items()))
'chs=500x600&cht=lc'

An earlier version of this example involved a lambda: «’&’.join(map(lambda item: ‘=’.join(item), d.items()))», but then just as I was pasting it into this article I realised I could drop the lambda altogether. Bound methods rule!

What Python have you learnt recently?

16 Responses to “I learn Python”

  1. Thomas Guest Says:

    I don’t use the “locals()” trick much either – but then I don’t often use the dict form of string formatting. In Python there’s generally a dict behind the scenes holding what you need.
    For more advanced string operations, I’ve discovered string.Template. The safe_substitute() method lets you build up a complex text structure in a lazy way, then finish off with a substitute() to fully evaluate this structure.
    Sorry, I don’t have a good example of what I’m talking about!

  2. Leif Johnson Says:

    I was going to complain about the use of map and .items in your last example (I prefer iterators whenever possible), but I tried out your code and my proposed alternative and found that yours is much faster! So that’s something I’ve learnt recently. :)

    ~$ python -mtimeit -s”d=dict(a=’1′,b=’2′)” “‘&’.join(map(‘=’.join, d.items()))”
    100000 loops, best of 3: 3.21 usec per loop

    ~$ python -mtimeit -s”d=dict(a=’1′,b=’2′)” “‘&’.join(‘=’.join(i) for i in d.iteritems())”
    100000 loops, best of 3: 7.49 usec per loop

    Also, the timeit module is quite helpful for performance micro-benchmarks, if you’ve never used it before.

  3. drj11 Says:

    @Leif: I like iterators too, and one day (Py3K?) .items() will produce an iterator. I note that map already works if you pass it an iterator.

    I’m certainly not going to complain if you prefer a generator expression to map. Well, not until you remove map from the language.

    timeit looks fun; I’m surprised there’s a significant speed difference.

  4. Paul Hankin Says:

    Lots of things are becoming generators in py3k that currently produce lists. map and range for sure (making itertools.imap and xrange redundant), and I’d guess dict.items too, although I’d have to check.

  5. Ian Bicking Says:

    urllib.urlencode(dict(a=’=’)) will give you a=%3D — your example isn’t doing proper parameter URL encoding.

  6. AdamG Says:

    Oh come on, there *isn’t* a significant speed difference :). 3 microseconds versus 7 microseconds? Don’t base your decision on that, base it on what you consider most readable given the context. Sometimes it’s map, sometimes it’s a generator comprehension, sometimes it’s a for loop. Trading N microseconds for reader comprehension is a good deal any day.


  7. “The optimisation I am talking about is one where «”some constant string” % stuff» gets converted into «some_function(stuff)» where some_function is a compiled function that does the formatting.”

    ”some constant string” % stuff

    Is equivalent to:

    str.__mod__(”some constant string”, stuff)

    You can use psyco to turn this into a nice compiled function call.


  8. which text editor is that ?

  9. drj11 Says:

    @Michael: And does it partially evaluate the function? For example, “%s%s” % p is equivalent to the function: lambda(p):str(p[0])+str([1]) which is what you’ll get in Lisp, the compiler actually inspects the format string.

    @AdamG: Agreed, and I never intended to suggest I would prefer one code over the other on the basis of speed.

    @Ian: Ah, thanks. You can tell I don’t write web-apps can’t you?


  10. I realised I could drop the lambda

    Ah, the autumnal smell of freshly-mown η-reduction!

    one day .items() will produce an iterator

    and map will be iteritems.imap, and everyone will be happy.


  11. I think, that there is no any sense, to replace d.iterm with d.iteritems, because there only view items in the dictionary. Iterator is an unnecessary overhead in this particular case.

  12. luma35 Says:

    Could tell me how did you code formatted like that???

  13. drj11 Says:

    @Luma35: I’m sorry, I don’t understand you. You appear to be asking something about code formatting, but I’m not sure what exactly.

  14. luma35 Says:

    I mean on your page there code is formatted with line numbers! How did you get that?

  15. drj11 Says:

    @luma35: Right. It’s a standard wordpress.com feature. See http://faq.wordpress.com/2007/09/03/how-do-i-post-source-code/


Leave a reply to drj11 Cancel reply