Tim Tyler writes a dry ironic piece titled «Destroy the Entertainment Industry» (warning: contains 2 tits). I can’t tell if he’s being genuinely serious or not. His arguments canter along at a nice pace, until suddenly he nukes himself in the foot, by suggesting germ-line manipulation as a solution. Specifically, “If humans exhibit variation in their susceptability to entertainment-induced pathologies, maybe a human can be constructed that is less prone to this type of malfunction”.

Of course, I’m not denying that there is a gene for “susceptability to entertainment-induced pathlogies” (pedants: “a gene for X” is used in the same sense as Dawkins uses it in «The Selfish Gene»), it may even be possible to manipulate the gene or its hosting population. The problem with this argument is that it’s so universally applicable so as to make it useless. Germ-line manipulation now becomes the crucial new tool in solving any public policy problem (or in fact, any problem at all). People blow each other up too much, that’s a problem. How do we solve it? Germ-line manipulation, simply eliminate the gene for “blowing each other up” from the population.

This turns into a rather amusing game, where when anyone poses a problem, you simply reply with your mantra of germ-line manipulation. Let’s see how it’s played:

People abuse the disabled parking spots in public car parks? Germ-line manipulation solves the problem by removing the gene for “being an asshole” from the population. Too much dog poop on the streets? Germ-line manipulation. Too many stupid blog articles? Germ-line manipulation.

So far (1 evening of play) it seems to be a really good game, it’s instantly relaxing and manages to be fairly compulsive and interesting. The water is done really well in several different ways. Underwater in the lagoon the water dims just the right amount to make it feel like a private pool. The droplets running off your visor and evaporating when you surface are just spot-on.

It makes me want to play Wave Race 64, which still has most excellent water. Available on the Wii virtual console apparently.

I’m just left wondering:

1) Why even a pacific diving simulation game has an annoying bint of a side-kick; and,
2) When it’s all going to be revealed that the fish are actually a new military genetically modified breeding program for aqua-metroids, and there’s a lost pirate research vessel buried on the ocean floor. In other words: When do I get my plasma cannon?

Cate Blanchett in jack boots, with a sword. Think on that for a moment or two. It has the potential to fulfil fantasies you never knew you had. With such promise how did it manage to go so wrong?

And that crystal skull was clearly a cheap plastic knock-off.

Recall that calloc allocates contiguous space for n objects of size s:

void *calloc(size_t n, size_t s);

Observe that this is the same as allocating a block of size n×s. So why not just go malloc(n*s) instead? Well, for one thing calloc initialises the memory to all-bits-zero, but I’m not concerned with that right now. The problem with malloc(n*s) is that the product n×s can overflow:

malloc(65537*65537)

On most 32-bit platforms this will overflow 32-bit arithmetic and be silently equivalent to malloc(131073). That’s a problem.

Because calloc is passed the factors of the product it can, in principle, test the factors to see if the product would overflow, before using the product. In the case where n×s would overflow it can simply fail the allocation (returning NULL) instead of allocating a block of completely the wrong size.

I don’t think the standard gives implementations much wiggle room in this area:

The calloc function allocates space for an array of nmemb objects, each of whose size
is size. The space is initialized to all bits zero.

Returns

The calloc function returns either a null pointer or a pointer to the allocated space.

“Returns either a null pointer or a pointer to the allocated space”, nothing about the behaviour being undefined if the implied required size exceeds size_t or anything like that. However, I don’t think any serious C programmer would think that calloc is required to catch the overflow case, certainly I didn’t expect it, and I think it would be a good thing.

So, what if you wanted to implement calloc so that it catches the overflow case and returns NULL. This boils down to:

Given n and s, each a size_t, how do you detect when the product n*s overflows?

Obviously I’m more interested in portable solutions than not.

After a brief bit of thinking about multi-word arithmetic and other such horribleness the following occurred to me:

if((size_t)-1 / n > s) {
    return NULL;
}

(note that (size_t)-1 is guaranteed to be the largest value that will fit in a size_t)

I like this code, it’s portable, it’s clear, it does the right thing. The only eensy-teensy leetle worry I have is that the division might be a bit slow. Since the allocation of memory can take as few as 20 instruction cycles (see [GRUNWALD 1993]), an additional 40 or so cycles for a division might be too much overhead sometimes. We could have a fast conservative test for the okay case. For example if both n and s are smaller than 65536 then we’re guaranteed to not overflow:

if((n > 65535 || s > 65535) &&
    (size_t)-1 / n > s)
{
    return NULL;
}

This avoids doing the division when both n and s are suitably small. Of course there are still a range of cases where the product n*s does not overflow, but we end up doing the division test anyway. That’s not so bad because in those cases we’re allocating at least 65536 bytes anyway.

The 65536 test is a bit weak. It’s the largest value we can portably test against because size_t’s maximum value might be as low as 65536²-1. Can we portably do any better? If we could compute square roots at compile time we could do something like this:

if((n > sqrt((size_t)-1) || s > sqrt((size_t)-1) &&
    (size_t)-1 / n > s)
{
    return NULL;
}

but I can’t think of way to do that.

What if we merely accommodate the next most common size for size_t, namely 64 bits? Observe that 2⁶⁴-1 is 641×6700417×4294967295 (the latter factor is 2³²-1, the first two factors are the prime factors of 2³²+1).

Here is a portable test for “size_t has at least 64 bits” that is a compile time constant:

((size_t)-1)/641u/6700417u >= 4294967295u

This is portable because all the literals are guaranteed to fit in unsigned long.

So we can use this test to select a lower bound for maximum size_t value’s square root:

#define LBMAXSZSQRT ((((size_t)-1)/641u/6700417u >= 4294967295u) ? \
    65535 : 4294967295)

if((n > LBMAXSZQRT || s > LBMAXSZSQRT) &&
    (size_t)-1 / n > s)
{
    return NULL;
}

We could extend this approach to support this sort of early-pass test for any finite number of size_t widths, but 32 and 64 seem to be the most important. In any case we still get some sort of early-pass test that avoids the divide whatever width size_t is.

So that’s how to make a nice robust calloc. We’ve all heard the blurb about how we should rely on the C library wherever possible because its routines will be highly tested, robust, and carefully optimised. So naturally every existing calloc out there performs this overflow check and doesn’t just multiply its arguments together? Oh no. Even I was surprised.

On OS X (10.4.11) calloc(65537, 65537) returns a non-NULL pointer. This is a bug.

PHK’s code in NetBSD is typical, it just multiples the two arguments and uses the result, reckless to overflow.

Hurray for FreeBSD: they do almost exactly what I suggest, but with a fast test that is just the tiniest bit unportable.

REFERENCES

[GRUNWALD 1993] “CustoMalloc: Efficient Synthesized Memory Allocators”; SOFTWARE—PRACTICE AND EXPERIENCE, VOL. 23(8), 851–869 (AUGUST 1993); 1993-08

As I was looking at ms I came across some C code of the form:

floor(1 + x) - 1

where x is a double precision floating point value. (I’m too polite to show you the actual code, see streec.c line 443 if you must).

Question: when is this different from floor(x)?

It’s quite a fun little puzzle. I give my answers below.

For the purposes of this discussion I’ll assume that double is an IEEE double precision floating point type. So that means double exactly represents numbers of the form

m×2^e

where m and e are both integers, respecting the following ranges:
-2⁵³ < m < 2⁵³
-1074 <= e <= 971

2⁵³ is the first value that occurred to me. This is the smallest integer representable as a double where its successor is not exactly representable. So if x is 2⁵³ then x+1 is also 2⁵³ (IEEE round-to-even), floor(2⁵³) is 2⁵³ (it’s an integer), and 2⁵³ - 1 is exactly representable (the successor of 2⁵³ is not exactly representable, but its predecessor is).

A similar thing happens for some of the numbers bigger than 2⁵³ and less than 2⁵⁴. In this range the step between adjacent doubles is 2 (in the representation scheme above, e is 1). These doubles are all integers, so floor isn’t going to change anything, so floor(1+x)-1) is (x+1)-1, and floor(x) is x. When x is a double represented by an m that is odd then x+1 and x will be different because of the round-to-even rule. For exactly the same reason x+1-1 will be the same as x+1. An example is x = 2⁵³+2. This is represented as 4503599627370497×2¹, but x+1-1 is 4503599627370498×2¹. In general a similar thing happens for a number of the form 2i where i is an odd positive integer and 2⁵² < i < 2⁵³.

The previous problem was that we had a number x and adding 1 to it made the result round up to the next higher integer. A similar thing happens for very small negative values. If x is negative and very close to 0 then x+1 might be sufficiently close to 1 to be rounded to 1. This happens when -2^-54 <= x < 0 (numbers just a bit less than -2^-54 get round to 1-2^-53 when 1 is added to them).

-0. This is sort of degenerate case of the above. -0 can be thought of as being just the tiniest bit less than 0, and this difference cancels completely when 1 is added.

In the above case when x is small and negative what happens is that it is just smaller than an integer, and this difference gets cancelled entirely when 1 is added. So that whilst x is not an integer x+1 is. A similar thing happens at every number that’s just a bit less than an integer power of 2 (a reasonably small positive power of 2). For example 7.999999999999999 (which is 8-2^-50) rounds to 9 when 1 is added. So floor(7.999999999999999+1)-1 is 8, but floor(7.999999999999999) is 7. This happens for every number of the form 2ⁱ-2^i-53 (this is nextafter(2ⁱ, 0)) where i is a non-negative integer and i < 52.

Possibly I’ve missed some cases. Especially negative ones.

In the case where I originally found the code I’m pretty sure the difference is irrelevant.

Here’s why tabs are wrong:

        if( (segsites > 0 ) || ( pars.mp.theta > 0.0 ) ) {
               if( (pars.mp.segsitesin > 0 ) && ( pars.mp.theta > 0.0 ))
                       fprintf(pf,"prob: %g\n", probss ) ;
           fprintf(pf,"segsites: %d\n",segsites);
                   if( segsites > 0 )   fprintf(pf,"positions: ");
                   for( i=0; i<segsites; i++)
              fprintf(pf,"%6.4lf ",posit[i] );
           fprintf(pf,"\n");
               if( segsites > 0 )
                  for(i=0;i<pars.cp.nsam; i++) { fprintf(pf,"%s\n", list[i] ); }
            }

I never want to see this shit. That was how this code looked when I opened it in vi. With tabstops set to the default of 8. It is of course much improved once I guess that the correct tabstop should be 4:

        if( (segsites > 0 ) || ( pars.mp.theta > 0.0 ) ) {
           if( (pars.mp.segsitesin > 0 ) && ( pars.mp.theta > 0.0 ))
               fprintf(pf,"prob: %g\n", probss ) ;
           fprintf(pf,"segsites: %d\n",segsites);
           if( segsites > 0 )   fprintf(pf,"positions: ");
           for( i=0; i<segsites; i++)
              fprintf(pf,"%6.4lf ",posit[i] );
           fprintf(pf,"\n");
           if( segsites > 0 )
              for(i=0;i<pars.cp.nsam; i++) { fprintf(pf,"%s\n", list[i] ); }
        }

but later down the same file (with tabstop still set to 4):

    }
   else if( segsitesin > 0 ) {

        pk = (double *)malloc((unsigned)(nsegs*sizeof(double)));
        ss = (int *)malloc((unsigned)(nsegs*sizeof(int)));
        if( (pk==NULL) || (ss==NULL) ) perror("malloc error. gensam.2");

      tt = 0.0 ;
      for( seg=0, k=0; k<nsegs; seg=seglst[seg].next, k++) {
        if( mfreq > 1 ) ndes_setup( seglst[seg].ptree, nsam );
        end = ( k<nsegs-1 ? seglst[seglst[seg].next].beg -1 : nsites-1 );
        start = seglst[seg].beg ;
        len = end - start + 1 ;
        tseg = len/(double)nsites ;
               if( mfreq == 1 ) pk[k] = ttime(seglst[seg].ptree,nsam)*tseg ;
               else pk[k] = ttimemf(seglst[seg].ptree,nsam, mfreq)*tseg ;
                 tt += pk[k] ;
      } 

which doesn’t look very tidy whatever I set tabstop to. Also, the curly ket preceding the else is to the left of the else and on an earlier line; that confuses the JavaScript syntax coloriser code (which isn’t mine) and causes it to crop the else token incorrectly. Select the “view plain” link to see all the code.

The code is from ms, a genetic sample generation program used by scientists.

It seems to me that tabs are an attractive nuisance. Something that seems like a good idea, but which can lead to trouble. Sure, you and I can use them safely, but in the hands of the péons they just lead to madness.

x&y is a long when either x or y is a long. When it could/should be only when both are long.

In particular x&1 is a long whenever x is. Despite the fact that the numerical answer is either 0 or 1.

I’m not saying this is wrong, just… ugly.

Roll on Py3K.

I also announced this on the Curly Logo news feed, but it’s sufficiently amusing that I’ll also mention it here. And also inject some discussion.

So, Curly Logo now works on the Wii. I think this is quite cool, and quite impractical. Of course there isn’t really a Wii edition, there’s one “edition” that all browsers and all platforms see.

So the Wii is a bit of a frivolous platform, and I don’t think I’m going to get very many Curly Logo users on the Wii. That’s not why I made Curly Logo work on the Wii. In fact it was only few weeks ago that it even occurred to me that Curly Logo might work on the Wii (I only used the Wii’s browser briefly after I first got it, mostly for the novelty of reading the BBC news website on my TV). At first I dismissed the Wii, silly platform, awkward UI, not worth it. But then I came round to thinking that the Wii represents a cornerstone of the Curly Logo philosophy. One that could be briefly described as minimal installation, maximal accessibility.

The reason Curly Logo is implemented in JavaScript and not .net or flash or C is so that it can be deployed to anyone using a browser, with zero installation. As few barriers as possible between you and using it. In a way I see this philosophy of being a descendent of 1980’s language implementations in ROM. So this philosophy is all very well and good, but to be of any use, I actually have to get Curly Logo working on random browsers. Which is why I decided to get Curly Logo working on the Wii. As a sort of proof of concept that it was reasonable to get JavaScript only applications running on more or less any suitable browser.

It turns out that the tricky part of getting Curly Logo running on the Wii was getting it running on Opera. And happily, the desktop version of Opera is sufficiently like the Wii version that once it was working on desktop Opera it just worked on the Wii. The Wii even implements SVG; who would’ve guessed?

Opera seems like a bit of a funny browser. It’s like they’re trying to be a full fledged standards compliant A-list browser, but they just don’t make the grade like Firefox and Safari. But perhaps that’s just because I spend more time with those two.

So the two speedbumps on the road to Opera glory were: btoa; and, CSS box-model. So it turns out that whilst Firefox and Safari implement the non-standard btoa function, Opera does not. Curly Logo uses btoa when it generates a 1×1 PNG file to use as the background for the text area. Happily I already had a JavaScript implementation of btoa that I used for testing. Parts of Curly Logo are tested using a command line JavaScript interpreter (SpiderMonkey) which doesn’t include btoa, so I had to write a version for that. It was a simple matter to move it from test code, which is not delivered, to product code.

The CSS box-model is probably the part of Curly Logo that I understand the least. I don’t think I’m trying to do anything heroic (after all there are only two boxes: the SVG drawing area, and the text area), it’s just the CSS spec is very complex, and I don’t understand it. The problem was that the drawing area containing the SVG was only tall enough to show the top half of the turtle. And I think this turned out to be because Opera had decided that the best height for the SVG element was just big enough to show half the turtle. I guess conceptually it’s a bit of a tricky problem, how big should the box be that contains a random load of SVG? Anyway, explicitly fixing the height of the SVG element to be the same as its containing block sorts it out (the CSS is: «svg { height: 100%; }»).

There are two more Opera quirks that I’ve noticed that I’ve left unfixed for now: settbg, to SET Text BackGround colour, doesn’t work unless you specify alpha, so settbg "#5c5c works just fine, but settbg "#5c5 borks and dumps a JavaScript exception backtrace all over the text area. Quality. The other quirk is that the arrow character used to represent evaluation, as in «2 * π ⇒ 6.283185307179586», displays as a box in Opera (on my Mac, on the Wii it works just fine). The settbg problem is caused by lack of atob (the inverse function of btoa), and I happen to not have a JavaScript version of that lying around. Yet. The arrow/box problem is inexcusable and not my fault. I suppose I could work around it by choosing a different character (or drawing the arrow in SVG!) but I’m not inclined to do that. The arrow convention is nicked from CLTL2, by the way.

For Curly Logo the Wii is not a great platform. The TV is pretty fuzzy, the text is small, the colours are wrong. Typing commands in the text box is awkward, the Wii screen keyboard completely replaces the web page so you can’t use any context. But it works. I think it vindicates the JavaScript “write once, run anywhere” philosophy. I wonder if the Wii implements the TV CSS media type, that would be the best way to fix the font size problem.

Speedwise Opera seems to be pretty nippy, it comes in somewhere between Safari and Firefox. For example the dragon takes about 15 seconds in Opera (Safari: 5, Firefox: 25). Even on the weedy little Wii (729 MHz PowerPC), dragon takes just over 2 minutes, which isn’t too much of a wait. The best bit is that there’s no “Click here to continue running this slow script” button to press, and that you can see the turtle in motion. The downside is that it didn’t take me very long before I crashed the Wii, requiring it to be reset with the power button.

PyConUK 2008 is asking for submissions. You should give a talk.

You should give a talk because I want to hear what you have to say. And so will others. People tend to think that no-one will want to hear what they’ve been doing, or that it won’t be interesting. But on the whole I just don’t think it’s true.

Communication is a skill, and it is improved by practice. I think computer scientists and software engineers should all strive to be better communicators.

Python is not a huge community, we do not have a PR department. If we want Python to grow then we have to grow it ourselves. Conferences like PyConUK are an important part of that community growth, and the speakers—that’s you—that provide content are an important part of the conference.

What should you speak about? Why you love Python, why you hate Python. Your favourite module. How you saved the world from global nuclear catastrophe with a simple Python script. Anything really.

I’ve submitted my talk already, and—shock—it’s not about functional programming.

Look forward to hearing what you have to say in September.

WordPress gives me a very convenient view of what people have been searching for when they reach my blog. I think most readers are regulars; some come from sporadic programming.reddit.com spikes, but some come via search.

I feel sorry for some of these people, coming across my blog in the middle of their search must be like stepping on a rake whilst looking for the lost ball in the garden.

«Element Not Good For Monks» - ok, so the search is a bit wacky, but I’m sure I’m not the blog they’re looking for. There’s a Kryptonite for monks?

«quilting pattern flower uk» - Search engines have obviously decided on the basis of this post about purple things, and this one about multiplication in rings that I have something to do with flower quilting patterns. Sorry, no.

«multiplication table» - this is a relatively popular search term, it crops up a few times a week. Sufficiently popular that I’m thinking of creating a blog post for it.

«c++ matrix multiplication code» - close, but no cigar. Anyway, using Google to do your homework assignments is cheating.

«how to find sex of foetus in ultrasound» - it’s clear what they’re looking for and my blog doesn’t help them at all. If I had to guess I’d say you look for the willy.

«emacs python color» - there are quite a few variations on this theme, obviously looking for emacs modes that do colour syntax highlighting for various languages (not just python, I see fortran, javascript, and so on). Sorry, no idea.

«metroid 2 map» - Well, I suppose I do have a Metroid II map, but really it’s a celebration of how not to do mapping. Sorry, not very useful. Anyway, regardless of what I think about maps in video games, using somebody else’s map is clearly cheating.

«where is the colon» - umm… somewhere behind your belly button?

«most fragile part of the brain» - I have no idea, and I’m not just about to experiment to find out.