Monty Hall, Monty Hell and Tuesday's Boy

Consider the following problem (which I've quoted from the wikipedia page):

Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

 Well. Is it?

Click here to read the rest

Bridge Probability

I've been re-reading Victor Mollo's classic bridge book 'Card Play Technique: the art of being lucky'. I've come, once again, to the sections about probabilities that I just can't get my head round. Consider the following passage:

If the Ace of Clubs is right, all is well. If not, the contract will depend on guessing the diamonds. How then, should we set about it? The man in the street will draw trumps quickly, sway in his chair slowly and mutter something like: "Well, it's six of one and half a dozen of the other".
But is it?
To the expert there is a vital difference between the Club and the Diamond positions. The latter will be the subject of guesswork. The former lends itslef only to prayer. The Club must be played first, and the reason is that it will provide a clue to the Diamonds.
If East has the Ace of Clubs, West will be credited with the Ace of Diamonds.

There is similar reasoning throughout the book, just one hand later:

West has pleaded guilty to 7 points hearts and to the King of Diamonds: 10 in total. East has only 3. It is more likely that the defender's high-card strength will be divided between them 10-5 than 12-3. Therefore the best chance is to play East for the King of Clubs.

I just can't believe that this reasoning is valid. The deck doesn't know which cards we assign points to, so it is no more likely to put the Ace of Clubs in a different hand to the Ace of Diamonds than it is to put, say, the Three of Clubs in a different hand to the Ace of Diamonds. Similarly, while it is true that a 10-5 distribution is more likely, a priori, than a 12-3 distribution, surely this ceases to be the case when you've alread placed the remaining points 10-3 (assuming that the principle of Vacant Spaces doesn't come into play - ie, every player has followed with a small card to each of his partner's honours).

To repeat: the deck doesn't know which cards we assign points to!

Maybe this sort of reasoning is a useful shortcut to some valid reasoning, or maybe I'm missing something (I certainly hope so). Can anyone shed any light?

Druids Demonstrate Regression to the Mean perfectly

I once heard of an experiment that a friend of mine (an anti-speed camera campaigner) used to do when he was giving a presentation. He would get everyone in the room to randomly generate a 2-digit number (not sure how, exactly, maybe he used to carry some sort of 10-sided dice with him). He would then give everyone who rolled more than 80 a big picture of a speed camera and got them to generate a second set of numbers. Lo and behold, only a small fraction (on average around 1/5) of those carrying speed camera signs had high levels of deaths. The speed cameras had worked!

This is the purest way I've ever seen to explain the phenomenon of 'regression to the mean'. It's a well-known phenomenon, and explains a lot of things, from why test scores at the worst schools tend to improve the next year, through why patients who visit a homeopath seem to feel better, to the Sports Illustrated Jinx.

However, the Austrian motorway authority seems never to have heard of it. They recently asked some druids to reduce the number of fatalities at a few accident blackspots by burying some magnetic slates. The results were a roaring success:

Austrian motorway authority ASFINAG said it was sceptical at first and kept the project a secret. But it went public after the druids’ efforts cut the number of deaths at the notorious crash site from six a year to zero in two years.

I don't think there's much more to say.

How to sample randomly

I was once told that 'duration of unemployment' figures were collected in the following way: people were telephoned at random during the day. If they answered, they were asked if they were unemployed. If they said yes, they were asked for how long they had been unemployed. Before you read any further, can you see what is horribly, horribly wrong with this method of data collection? (There are several things wrong with it, but one of them renders it entirely useless)

Before I give you the answer, a brief detour. When I was at school, I did a piece of statistics coursework (I think it was for GCSE's) in which I compared the average sentence length in a French text to an English text. I can't remember exactly which texts I chose, I think it was newspapers of 'equivalent' quality, but that's largely irrelevant. In order to estimate the average length of sentences in each text, I adopted the following method: pick a word uniformly at random from the text and count the number of words in the sentence containing it.

I had collected around 100 sentence lengths before I noticed the utter ridiculousness of this method. In case anyone hasn't spotted it yet, this 'random sampling' is guaranteed to massively overestimate the average sentence length in any given document, as the probability of any given sentence being chosen is in direct proportion to its length.

Consider the following passage:

"The quick brown fox jumps over the lazy dog whilst the five boxing wizards jump quickly over my lovely sphinx of quartz. Jesus wept"

 If we pick a few random words from this and compute the 'average' sentence length of the sentences that contain them, we're going to come up with something very close to 20 (if we pick every single word, we'll get 20.3333)  The actual average sentence length is 12.

Now, if you didn't immediately spot that this was the key problem with the method of collecting unemployment data I mentioned in the first paragraph (there are problems with telephone polls in general, of course, but they are essentially insignificant compared to the problem with the sampling method), this should make you worry about how easy it is to slip *exceedingly* dodgy statistics past people who aren't paying attention. I'll post a few examples of my favourite 'correlated for spurious reasons' statistics in another post later this week.

As an aside - if you do actually collect the data in the way suggested, you can presumably still get some information about the distribution you're studying - what's your best estimator for the mean? And what assumptions do you have to make about how the data are distributed?

Why don't we sample more?

Steve Landsburg recently blogged about a maths professor who weeds out 'unlucky' applicants by randomly rejecting half of the resumes he gets sent. Now, this is unusual, in that it is a random sampling method which significantly *reduces* the average quality of the applicant that gets hired.

There are a *lot* of situations in which random sampling would reduce workload whilst having no effect whatsoever on effectiveness. I'll start with one of the simplest and least controversial (and one that I have the most personal experience with). Students regularly submit 10 or more pieces of coursework for each course in a university semester. Every question is then marked, and the papers returned to the students. Assuming (which is probably not entirely accurate) that the courseworks are solely intended as a normative assessment of student performance, surely it would be massively more efficient to sample questions at random and mark those, rather than marking the entire paper. The expected mark for any given student is the same - only the variance goes up.

There are a few situations in which students suffer as a result of this. Say there's a pass mark of 40, and you have to pass every coursework, now someone who answers exactly 40% of the questions right in each coursework expects to fail (although they do expect to get an average mark of 40). Similarly, there are situations in which students benefit from this (pass mark of 40, answer exactly 39% of the questions correctly, you now have a non-zero chance of passing). On the whole, I would expect these things to cancel out, and that no one student knows their mark accurately enough to know whether they would benefit or lose out from this policy being enacted.

So why isn't this done more? I've heard from a few lecturers who've tried it, and it went down horribly with the students, who perceive it as 'unfair'. Apparently there were several comments along the lines of 'what if you only mark the questions I did badly?'. I guess this is some sort of loss aversion - it is quite obviously equally likely that we only mark the questions you did well!

Yvain has an article about a similar example from education - in which students are reluctant to guess answers to true/false questions with a penalty of 50% of a point for a wrong answer for some inexplicable reason. Again,  random sampling is a massive net win.

Another example is public transport. No-one every pays to get on the 25 bus. This is because it is extremely rare for anyone to check whether you've paid or not and the penalties just aren't high enough to make it worthwhile paying given how rare the checks are. There are two obvious solutions to this problem: you could either do twice as many checks (thus requiring you to hire twice as many people to do the checking, and inconvenience twice as many people whilst checking) or you could double the fine. I've no idea why they don't take the second option.

How about voting? Instead of counting all of the votes in a general election, why not shake the votes up in a big bowl and count, say, the first 10,000 for any given seat? I can't be bothered to crunch the numbers, but I'm pretty sure the probability of error would be down below 1% - and errors would only occur in seats which were closely contested - where errors are not so important anyway, as the people obviously don't have a clear preference between the candidates.

Most of the examples I can think of exploit the same principle as the public transport idea above - when committing some transgression, your expected utility is the utility of cheating minus the disutility of punishment times the chance of getting caught. Since it's expensive to increase the chance of getting caught, there are a lot of situations in which I think it would be a net win to decrease this and increase the size of the punishment. Why not check half as many tax returns and double the fine for misfiling? Have half as many speed cameras and double the fine for speeding (speed cameras aren't expensive, so this might not be a net win)?

There are dozens of examples - and I don't think that the people in charge have sat down and done the relevant calculation in all cases. Are people just afraid of randomness? Afraid of seeming 'arbitrary'? Afraid of letting people 'get away with' committing crimes - assuming the only legitimate purpose of the criminal justice system is deterrence, this shouldn't be an issue. Maybe there's legitimate concerns that a 'random sampling' approach to some of these problems would be more subject to corruption - but we can just check a few of the samplers at random, and have massive fines for people doing it corruptly!

The law of large numbers is a powerful and important mathematical theorem. Why don't we exploit it better?

Gladwell on probability

There's quite a nice list of random quotes from Malcolm Gladwell in an interview for this Sunday's Observer.

However, one of them seems to show some misunderstanding of probability:

History suggests that there is almost exactly a 50% chance that any piece of information a spy gives you is true. We would be as well off getting rid of the secret service and flipping coins.

Now if the first part of this sentence is true (which I have no reason to doubt) the second part most definitely does not follow. This is (tangentially) related to a discussion that's been going on at Peter Cameron's blog about probability. Unless spies only ever make statements about things where your prior was already 50%, a 50% accuracy rate could be incredibly useful.

Eg, let's say we're trying to find out where a particular terrorist group has their headquarters. To start with, our probabilities are essentially uniformly distributed across the whole of the world. Our spy comes up to us and says 'the HQ is at number 32 Barkston Gardens, Earl's Court, London'. This information is far from useless - in fact, if we have more than one spy coincide on the same piece of information then we're in business, and can find the location pretty quickly.

Of course, I think Gladwell's '50%' is actually just a proxy for 'exactly as true as you'd expect if they were generating their statements at random', but that's not *quite* the same thing